When I thought about it, redux is a state store, so everything that’s in minimongo should be in redux.
Imho that’s kinda like saying “Pascal is a programming language, so Meteor should be written in Pascal”. Single fact that Redux is made to solve one problem doesn’t mean it should be the ultimate solution, because there are others too.
I’d say you’re correct. I’d also say that if you’re not hellbent on native NPM, Meteor still provides many times more than you just built.
Rx.spawn(): RxJS/generators.md at master · Reactive-Extensions/RxJS · GitHub
Either way, you’re spot on, NPM has been slowly eroding Meteor’s full stack dominance day by day. I think we’ll join the main branch of NPM, so to speak, soon enough that it won’t matter too much. In the mean time, there’s plenty of things to focus on and build so you’re not like “ah shucks, i wish I had webpack.”
I’m looking at what you built up and down. It actually looks like you have answers to most my points above, minus the async vs. fibers stuff. What about collection permissions? What is your checklist for what you need to truly compete with meteor?
This looks cool, very well connected! I’m curious about how much it can scale, have you done some benchmarks?
Fibers was a stopgap that went away in node 0.11 or 0.12 (that’s chiefly why Meteor is stuck on 0.10 I believe). We now have promises. Promises allow for generators. Generators allow for async/await functions. This provides something very similar to Meteor.wrapAysync. Callback hell is sooo 2013 
https://github.com/mattkrick/meatier/blob/fb530c82aef3cc06971e4e5f9a9d8fc79734965d/src/server/controllers/auth.js#L47-L56
The next step is an observer function (not to be confused with Object.observe, that won’t be moving forward) that uses an async inside a generator. Netflix has some good patterns regarding this. See https://github.com/jhusain/asyncgenerator. I’m really excited to see where this takes evented programming.
The web-dev world outside of meteor calls this optimistic UI. When a request is made, it starts savings every action after that request in a queue. Then, if the action comes back successful, it can dump the queue. If the action fails, it’s real simple to undo all the actions that accrued. Minimongo can’t pause the state. If a document comes in from another user & breaks your UI, minimongo can’t “undo” that add for debugging. redux can.
Permissions is handled in socket middleware. Before a subscribe is allowed, it must pass any tests you set out for it. https://github.com/mattkrick/meatier/blob/master/src/server/publish/subscribeMiddleware.js#L4
This is pretty close to a Meteor.publish The JWT acts like Meteor.user() in a publish function.
If you don’t need webpack, stick to Meteor, this obviously isn’t for you. Enjoy your initial page load times 
Goodness me, I’m not trying to compete with Meteor. Meteor is super simple & nothing beats it when it comes to a MVP. This is geared towards folks who are debating whether they have to dump node in favor of phoenix because meteor can’t keep up.
Not yet, I think first we need to come up with a legit benchmarking standard for sockets.
90% of benchmarks I see online answer the question, “How fast is X at doing nothing?”. See http://hueniverse.com/2014/08/20/performance-at-rest/ and tell me it doesn’t ring true.
Another consideration is not using a socket at all. If a certain Meteor app supports 1000 concurrent users and 900 of them are on the landing page reading the FAQ, congratulations, you’ve opened 900 sockets for the sole purpose of playing DDP ping pong. Isn’t it better to open a socket when the user actually needs it? How do you benchmark sound architectural decisions?
I’m guessing you don’t use redux ;). How do you edit your local state? If a new document gets pushed to your minimongo, how do you know? Using Collection.observe for debugging gets old real fast. If 100 docs come down the wire & 1 breaks your UI, can you replay the events 1-by-1? If you’re fine-tuning your animations and want to reply the event of a document being added optimistically, or an optimistic add being verified by the server, how do you do it? If your client emails you about an error, how do you replicate their state? Try redux, you might like it
If it sounds crazy, keep in mind this is how facebook’s relay works (with the original flux flavor). The difference is GraphQL returns joined docs, so they normalize the subdocs in the store in order to simplify mutations. There are even relay-esque projects built for redux like https://github.com/gyzerok/adrenaline. The problem is that these still can’t answer the question of how to handle a realtime connection & not just use intelligent long polling.
Since facebook has no interest in using sockets, the answer won’t come from them, it’s gotta come from someone else in the opensource community. This is where I think MDG can carve out their niche (just like @arunoda said in Why Meteor needs a new Data Layer?) If MDG pivots to focusing purely on the “data layer” then they can keep their galaxy & consulting business model while providing a realtime solution that appeals to the entire node community, not just a subset.
…but I’m impatient.
No, you are trying to beat meteor. To make something competitive. So own it–what needs to be done to be truly competitive with Meteor?
If you aren’t, you should. In the repo readme you listed several competitors that fall short. Mainly cuz they didn’t have the rethinkdb database layer or if they did they didn’t have the web server socket aspect as on point. So if you truly are ahead of the pack of other “isomorphic frameworks in the making for NPM” then why not aim to achieve just that goal. I do find it kind of hard to believe that there are so few such packages, but I’ve researched a few times myself and haven’t found anything that comes as close to competing with meteor in one all in one package as yours. I say you go for it. Carve out this niche. The pure NPM option for what you can do with meteor.
I also think you should make a second package that wraps all your method calls with a facade that has the exact interface as meteor. With the goal of making it so someone could switch their meteor app to yours and it would work. I’m pretty sure mongo is just a subset of what you can do with rethink. So for all the find, insert etc calls just make a thin wrapper over rethinkdb with the mongo interface, for example. That may already exist anyway and/or may be worth releasing independently. But you get the idea–mimic the meteor interface. Use react not blaze–so u don’t gotta worry about that one. Combine redux and minimongo in one way or another. I got a lot of ideas for that.
This will serve multiple goals–not just so people can switch out the underlying framework, but serve as a tangible checklist of what you successfully rebuilt. I’m not super worried if there are some areas that you can’t replicate. Eg the redux vs minimongo stuff. This would highlight the differences, and for those using redux in meteor already would result in no difference. But the point is if u successfully get far down this path, such things could be solved if we really wanted. Just grab tracker and minimongo packages independently of the rest of meteor. Maybe session too. That way meteor developers will still be able to do things like use my TrackerReact package which gives you enhanced reactivity in your react components: http://github.com/ultimatejs/tracker-react . I know ur all about redux’s time traveling debugger, but snapshots of all of minimongo and session would be the same thing and completely possible. Something I plan to build eventually.
Tracker offers an ease of development that react and redux doesn’t. Check my TrackerReact mixin. I agree everything going on in the react community is a big deal once apps reach a certain level of complexity. But tracker is still magical. So what I’m saying is that if you built this framework, and God forbid, to complete the goal of building a meteor shim on top of it, you had to incorporate minimongo and tracker, it would be a welcome addition for many. It also would only have to optional packages in your ecosystem anyway. Effectively you’d be making the unbundling of meteor a true use case for the first time. Blaze was hardly ever used outside of meteor for example. To unbundle something, there needs to be better avenues to use the sub bundles. Truth is, though MDG won’t be happy at the idea, this needs to happen to keep meteor on its toes and push meteor into the future. The future where they can’t be separate from NPM anymore if they have any chance of surviving as anything more than a niche offering. I’m with you on most your points about how easy it is to do most things with meteor.
CONCLUSION: why cop out now. U basically said u built what appears to be the 80% of meteor most used. Create the complete list of all the things remaining. The remaining things will likely be a lot smaller and nuanced and perhaps a very long list that in sum are less important than the first 80%. But let’s at least write it down to look at, so we know what we are dealing with. As it is you initially presented this as something that can truly compete with meteor (while also kicking its ass in some areas). So let’s take that thought all the way and not backdown now. No “pitchforks” here.
Ps. I booted it up by the way, added a bunch of lanes, time traveled with redux, viewed it in react dev tools, analyzed the code a bit etc. I was anticipating maybe a login/signup bug, perhaps on refresh. Nada. Everything was silky smooth. Nice work! This got potential as its own project. At least get a nice documentation site up before u move on to other projects that utilize it. Do a bit of marketing on Echo js, hacker news etc. Let the NPM community decide how important it is to them. I’d love to help myself.
Hahaha glad you find it useful
I’d love your help to get it out there & get some feedback, I think it’s a neat pattern. Let me remind you though I didn’t make anything, I just found some nice packages & put em all together. The beautiful people of the npm community did the tough stuff.
Regarding Tracker, yes, it’s easy to build, but it sucks to troubleshoot. It’s kinda like a polyfill for Object.observe, but that’s not the way forward. See https://esdiscuss.org/topic/an-update-on-object-observe. Two-way data binding is hard to reason about, react only uses 1 way data binding, so in that regard, adding tracker would be a step backwards.
The same goes for mixins: react ditched them because they don’t align to TC39’s vision for ES. But after writing a HOC or 2, you realize you don’t need em. That’s how I can deliver a websocket to a specific component: https://github.com/mattkrick/redux-socket-cluster/blob/master/src/index.js#L104-L142
Regarding mongo, the only thing that has a hard dependency on the DB is the message broker. The client doesn’t care what database you use, nor does most of the server. As long as the message queue can pull (or the db can push) mutations to be published, everybody’s happy. That means you could swap in postgresql in about an hour. If you wanted to scale horizontally, just move the message queue to something like redis or rabbitmq and you could host a very large number of concurrent users. See https://blog.rotenberg.io/million-user-webchat-with-full-stack-flux-react-redis-and-postgresql/
But that’s another $25 on compose.io. I’m not made of money (anyone hiring?). 
Regarding minimongo, RethinkDB is coming out with a client interface in the next month or 2, and then after that they’re talking about a GraphQL connector. Stay tuned: https://github.com/rethinkdb/rethinkdb/issues/3711#issuecomment-151639571
That’s the beauty of it, my friend! That’s the future NPM is all about–everyone building pieces of the puzzle, making layers swappable, swapping one layer out for another, aggregating/combining many packages into one layer, which can then be swapped out by better smaller pieces or a better large piece that includes it as one of its packages. With WebAssembly, even JS has a real chance of being truly switched out: http://www.sitepoint.com/future-programming-webassembly-life-after-javascript/
So basically, if you’re smart, you aren’t “building” anything per se anymore. And that’s the true danger Meteor is in. It’s the true danger any open-source commercial business is in. It’s killing the business model. To me, it’s far less smart of a business model than it has been the past few years, especially combined with how easy infrastructure has become as well. The smarter approach is to find something other than a licensed version with closed-source features or even an infrastructure play like “Galaxy”–the most noteworthy of which being “community,” i.e. like what Github and Facebook sell. Whatever closed-source features you build, someone else will build the open source version, learning from what works about yours. Whatever infrastructure offering/optimization you offer will soon be replicated for direct use on AWS.
I predict this next year will see this “cambrian javascript explosion” slow down. SLOW DOWN? Cuz it will have done its job. It will transmorph into something else. Our dream of completely solving the “native problem” without leaky abstractions will be a closed chapter (thanks to React Native). Realtime will be a closed chapter as well (RethinkDB, GraphQL). The reactive view layer already is (Redux). Debugging is on the way (not really thanks to Redux, but that gets a lot of people thinking), but due to the fact that Node-Inspector will likely be replaced by deep integration of dev tools by Node itself. Testing is closing as well as projects like “jsdom” improve and replace PhandomJS. So within a year, it will be a given that basically every layer is replaceable, and all the main problems are solved. We’ll now want new stuff. And likely, at this point we’ll be able to finally tackle the Bret Victor “observability” problem, etc.
So let’s definitely not get encumbered by doing any one package ourselves from the ground up. Well, lets try to avoid, you get the idea. …on a Side note–i dont know where it fits in the layer cake–but https://tonicdev.com is awesome. You’ll like learning how they take snapshots of state: Time Traveling in Node.js Notebooks …I wanna do something similar for minimongo and server state. Redux’s fatal flaw is that if the db changes state, you can’t truly replay state at a certain point, once its accessed again. That’s why the entire “isomorphic world” needs to be snapshotted. Their technology is able to snapshot opaque values in closures! Which would be an inevitable pitfall of a basic client-side mini-mongo + other objects snapshotter. If we could bring that technology to the browser (rather than just node), perhaps via a Chrome extension, we could do some serious damage. We could insure that all the redux time traveling stuff isn’t limited just to purely functional programming, which, yes, is a deal-breaker–I dont think relying on ur entire project being unidirectional + immutable will become the panacea people hope; it’s just going to be really hard for many types of projects to live up to that, probably the vast majority. Though, for plain js, Cycle.js is sick!
you don’t need 2 way data binding anyway. it’s more hype than anything, since you 99% of the time want to process the data entered by a user, and you can do something like model.populate() to get the data out of the form. To me, 2 way binding was nothing more than pre-built automatic “keyup” handlers. You can automate that without the need for true 2 way binding.
…As for Tracker, I’m not sure what you mean by 2 way binding?? I didn’t realize it offered that out the box. Enlighten me.
I didn’t think they “ditched” them. I just thought the class-based version didn’t support them. I don’t see why they dont just make the JSX transpiler automatically mixin an array returned from a mixins() method. After all, their mixin feature in the non-class-based version is nothing more than a combination of _.extend(component, mixin) + appending any lifecycle hooks in mixin to the event event emitter’s list of handlers.
…or use a component factory like you did.
I don’t follow PostgreSQL. Is the idea that triggers and procedures allow for observing queries? To what extent do they have built-in features for observing/subscriptions? Community-supported extensions??
Yea, that’s why minimongo may continue to be a core tool. Or perhaps rethink’s. And perhaps we are just making all our queries in GraphQL anyway, so we use what is built for now to actually store the data. When something better like rethink’s reaches maturity we replace it.
ps. great link on RethinkDB + Client Interface + GraphQL!
pps. i look forward to reading the “flux over the wire” article 
I can’t take anyone seriously who talks about the death of JS (and then says they like coffeescript in the next line). We all know JS came into this world like a premature baby, but we can’t replace it. The next best thing is employing really smart people to make it fast and easy to write. Every year they can add the best parts of all the other languages. Call me shortsighted, but I don’t think it’s going anywhere.
Yep, mixins are dead: https://github.com/facebook/react/issues/1380#issue-31121026
Postgres has NOTIFY and LISTEN, and also long polling. So you could rewrite a message broker to ask the database if it has anything new. Something neat with rethinkdb is that you can squash document changes. So, for example, if i have a 5 second squash, it’ll compile all the changes received for that doc over 5 seconds & then send it out. If a certain subscription doesn’t need to-the-millisecond reactivity, and the bottleneck is too many messages, this is the easiest scaling feature ever.
In my opinion, changing databases should require no change to the client. It should only know what data it needs. Someone will discover how to do this in realtime. I hope it’s MDG.
That guy is also overly against inheritance, when in fact that’s basically what you’re doing when you create React components. Fine, the implementation is becoming some custom factory, but to end users thinking of it as a “class system” is probably the “least cognitive burden.” However, if you’re still using the prototype-chain, you’re basically dealing with inheritance/classes. …I personally hate coffeescript. …I wouldn’t say “we can’t replace [javascript].” That guy is a bit overzealous. But everything gets replaced eventually, and Web Assembly would make that possible. The point is always to focus on what blocks need most replacing in the layer cake (and are easiest, i.e. who’s time has come), but the point is also that nothing is safe from being switched out. And that’s a beautiful thing, which the architecture and patterns provided by NPM have allowed to flourish more than ever.
…I do like the React team’s idea of turning mixins into a prototype-based inheritance system (stacked in between any standard inheritance hierarchy you have). I don’t know why they don’t just transpile calls to super automatically so you don’t have to write it. I guess so you can control if super is called before or after your code.
That said, their new mixin concept was proposed a year and a half ago and still nothing. Even then, it’s just a custom mixin concept using the prototype chain. I wouldn’t say–if they implement this–that mixins are dead, just enhanced/upgraded and optimized.
I don’t see how their factory under the hood can’t turn the “old n busted way”
class C extends React.Component {
mixins() { //yea, i know, you can't use mixins in ES6, classes, but I don't see why they dont transpile it for you, just as they should also transpile the binding of `this` in es6 classes, kinda like what is suggested here (in non-transpilation form): https://github.com/facebook/react/issues/1380#issuecomment-78216488
return [A, B];
}
}
into the “new hotness”:
class C extends mixin(A, B) {
…at least for backwards compatibility. And then only require mixin-authors to add super, or perhaps (just for legacy mixins) transpile it automatically at the top or bottom of the method for old mixins, so non-lifecycle (non “whitelist”) methods don’t continue to conflict.
ps. i have an inheritance system of my own now that operates exactly like React’s current one (I use it for things other than components), i.e. where lifecycle hooks are whitelisted and all run, while other methods overwrite each other–I’m tempted to “essentially create a new prototype chain with each mixin’s set of functions stacked on top of each other in order” instead. Solid sleuthing as always!
I remember i read somewhere that react might automatically do the super for you as well as the binding (eg this.onSubmit = this.onSubmit.bind(this) in an upcoming version. Here’s hoping!
Thanks! Don’t be shy about reporting any security or performance problems you find, after I get SSR & code splitting figured out it should be a pretty solid option for production apps.
@mattkrick: This is really nice work and definitely on its way to being a viable alternative. The Webpack build system and hot reloading are just a pleasure to use.
The main place I find the Meteor experience better, is its clear separation of framework code from app code. Meteor provides a lot of glue that stays mostly out of the way. Of course, there’s a good chance that what I’m seeing is just an application structure that I’m not used to.
You’re spot on in your assessment. That’s what makes Meteor awesome, you get some “good enough” defaults that are abstracted away from your business logic. This is great… until you need more than “good enough”. Different DB? Different auth strategy/workflow? An example: when someone tries to log in from the signup page, I like to say, “hey you nutcake, that wasn’t the log in page, but I logged you in anyways.” That’s simply not possible unless you expose the auth internals. It goes back to the main goal: trade a little magic for a lot of flexibility.
A more serious example is the pubsub logic. RethinkDB v2.2 introduced something called includeInitial that is very close to Collection.observeChanges(): it publishes all queried docs & then keeps the cursor open to push changes as they come. That means every subscriber gets their own channel & while RethinkDB can dedupe docs sent from the DB to the node server (or DB proxy if you’re scaling horizontally) it still gets expensive. Maybe you don’t need realtime for certain things. For example, no need to constantly watch for a username change, instead I can create a redux event like UPDATE_USER and after that makes its way to the server, I let the next longpoll catch it.
Hope that makes sense. If you see something that could be abstracted into its own package let me know! Ultimately, I think it’d be sweet to declare your business needs & let the internal system decide how to meet the constraints. Something like:
Meteor.subscribe(...,{
maximumUpdateDelay: 0, //use a socket, create changefeed, don't squash RethinkDB changes
maximumUpdateDelay: 120, //use a socket if open, don't use a changefeed
})
But I’ll let Meteor figure that out for their version 2.0
Hey, really nice work, @mattkrick and thanks a lot for the discussion, @faceyspacey.
If you implement a React Native Part like https://github.com/benoitvallon/react-native-nw-react-calculator, you are godlike
How much do you think DDP is a bottleneck for meteor? what if you are only using user auth as the reactive part (lets say compare to express app) ?
DDP isn’t a bottleneck, it’s just a protocol. In super basic terms its something like "when i say ping, you say pong. etc. Unfortunately, it never caught on in the realtime community.
I think instead of DDP you’re referring to using a socket for auth. IMO, user auth doesn’t need a socket because it rarely changes & the change rarely propagate to multiple clients (eg if user is logged on via phone & ipad). Honestly, how often will profile.lastName really change? Will your UX suffer if a 2nd tab doesn’t update the last name in realtime when it was edited on the first tab? If so, give profile its own collection & make that reactive, or even longpoll every x seconds to check the user doc. Meteor solves this problem by magically socketizing all the things. I chose to forgo this magic for some extra flexibility.
If you’re using Meteor & all you need is auth, you’re doing it wrong. It’s like driving a hummer to the grocery store.
sure, but how much actual bottleneck is it for just opening those websocket connections for user auth? I didnt pick meteor for realtime, i picked it for easy and fast development. (i also decoupled front and backend so i can always just switch out meteor for something better suited)