A suggestion for what Meteor v2 could be (RFC)

[spiralis]

I have been following the Meteor project from it’s early days. In the beginning the attraction was in that you could:

1. Do everything in javascript
2. Avoid the node waterfall callback coding style
3. The reactive data that was rendered in spacebar templates

From back then things have changed dramatically in the javascript world. The javascript implementations have matured to ES6/7 and the import/require system avoids the global namespace issue, allowing modular construction of the applications. Also, In the same time amazing things have happened in regards to libraries and the npm system. Due to this javascript rich client side applications have thrived and is now becoming the defacto web development practice. At the same time React and Angular had had great momentum. Personally I feel that the React way of doing things is the most important change in regards to finally being able to create reusable components. The next thing was seeing how Redux can help decouple applications, and how it in co-operation with React can help build applications that scale.

Looking at the divergence on Meteors v1 solution and the market out there it was evident that MDG had to do something. If they didn’t do something Meteor would die. And that would be a shame, because Meteor in itself and especially some parts of it really deserves the right to live. So, I applaud them in their attempt (and quest) to make Meteor stay attractive and alive.

Now, in 1.3 and 1.4 we are looking at the new future, but there still seems to be some uncertainty on where to go from here. I am currently building an application and using Meteor (1.3.4.1). I am using React and have adapted the application directory structure to the new recommendations. I loved this part as it made the application a lot more modular and removed a lot of dependencies. While doing this I was on my way to add Redux to the mix, when the question hit me like so many before me on how to integrate the database collection changes into the Redux state. Without having them there the option of properly logging and inspecting the timeline goes away. Also, having the collection changes reactive via minimongo directly to the DOM means that these changes have no traceability neither. I was looking at alternatives and the typical choices were to either let redux handle application state and let the domain state (collection data) stay reactive, or to add the reactive part into the state. The latter is the most clean way to do it, as that allows the historical debugging and the timeline. However, the latter option also means that the data will not only be stored in Minimongo, it will also be stored in the state. I then came across Cerebral, and it fealt like an epiphany in regards to how apps should be made. Granted, it looks like “another” Redux, and yes it is. But, it highlighted an important fact for me: The application itself can be made to run in its entirety without a view layer. Just using the console to drive actions is enough. Also, Cerebral has a pretty nice event timeline historical debugger (Chrome extension).

But, Cerebral in itself is not my choice though. It doesn’t solve reactive data-sources and easy collections (= rapid prototyping).

Now. I think Meteor should continue trying to make “application development” easy and simple, also in the future. And to do that I suggest the following principles:

1. Embrace developing apps via a central application state manager (ASM). This state may even also be totally handled server side too?
2. Let the ASM actions ask for data (like a subscription and a query in one). Changes in the data-sources are then logged as state-changes, just like manual user UI state-changes. The server-side data-changes are given via DDP to the ASM, which then handles the changes as an action (the change is coming due to an active subscription, so it is expected that someone is there to receive the changes).
3. Let the visual components be separate to the “application”. These are parts of the view, but only implicitly. Nothing in the components should be wired to the application. Note: Some frameworks does not have components, so this is then optional.
4. Have the view part of the application be handled in a separate part. Here the developer can set up the view using any tech / framework of choice. This part lays out the dom elements to the page and wires them up to receive application state.
5. Then components subscribe to change-events for the ASM states they want to monito. If we follow React philosophy we would subscribe to the root state, so that the complete application state is sent to the root state on any state-changes. React would then render the view in it’s virtual DOM before re-rendering changes only to the browser DOM. As an alternative solution (more efficient, but not as pure as React philosophy recommends), each component can subscribe for whatever leaf-state they are interested in. This way they will be notified only for the changes it is interested in. Now, this may or may not be your preference. The point is that as a developer you can choose.

This means that Meteor’s responsibilities will be:

1. Server-side:
2. Startup
   * Setup data-sources
   * Distribute startup files to the client.
3. Runtime
   * Receive and setup subscription queries
   * Send subscription query change responses via DDP to the ASM
4. Client-side
5. Startup
   * Setup the application Actions and States.
   * Setup the View, subscribe to state-change events (and implement view updates according to change-events)
6. Runtime
   * Handle UI changes via actions to the ASM
   * Act on data-source change callbacks from the ASM

This is a drawing that illustrates the application modules (not including the distribution of the initial files, the hot changes or server/client side script inclusion).:

Testing the application logic/state itself is very easy with this system. You don’t even need to start the view-layer. You can just run actions and verify the application state. for data-driven actions, these could be injection mocked.

I believe that Meteor would be awesome if something like this was supported. It would allow for setting up logics without considering the view-layer. It allows for re-implementing the presentation, without affecting the application logics. It is afaict presentation framework agnostic so it should attract anyone. The build-part of Meteor might have additional options for aligning with NPM and NodeJs(?). Testing application logics become trivial. Data is integral with state and feels “natural”, and includes Meteor DDP and reactive changes. I can even envision a nice bug-reporting system that collects the X last states and appends these to the report to help the for the devs to replicate the problem.

I think that this seems like a pretty good solution for me. Not sure if this is in contrast with the goals of MDG though.

Any thoughts?

[spiralis]

An additional note:

The application business logic should be modular too (like in the Cerebral modules). Meaning that a module will define both State and Actions. Some of these modules might be handy to reuse too (like common user handling scenarios) so they should support import and exports as well.

Probably also should mention that the subscription actions are “special”, since they are not actions that directly change the state. They are instead async calls that asks for data according to the subscription query is returned initially and then when changed. When the data is updated these will create a data-driven action that will then change the state.

BTW: Cerebral uses the notion of signals whereas I am coining this as actions. I am trying to keep the system as simple as possible and I believe that State and Actions are easy to understand. I also believe that using the pub/sub pattern for the scenarios should be well known and understandable.

None of the above is implemented and it might have conceptual flaws. That is why I am pitching it here. It is not that far from being possible to do with current libraries and Meteor though.

[seba]

I don’t really see the difference between celebral and the current minimongo system.
You have minimongo as central state manager. And you have define actions to change that state. You can already choose your frontend (react, blaze or angular), make it component-based and have container components so your core components don’t have to know about subscriptions etc. Instead of subscribing to change events, they subscribe to a query, which is basically the same thing.

It’s just that if I look to your schema, it already looks like how a modern Meteor 1.3 app is being build… Except for the different datasources and that’s where Apollo comes into play. Apollo should also allow you to do query & subscription in one, like you mentioned.

[spiralis]

Hm. That is somewhat true. I have never really thought of minimongo being that state-manager. I have always considered it the sync-point for the serverside data. The only thing that one would loose, which could potentially be a big thing for React purists, is that sending full state might be a bit more complicated to make sure that the non-changed items are still the old objects. Otherwise React components will unnecessary render in the virtual DOM.

But, I really like your observation on minimongo being this state manager.

An alternative approach to promote this way of doing things would be to add some syntactic sugar for setting up the states and the actions. And maybe the syntactic sugar would do some work for integrity in regards to what has been declared and the runtime minimongo changes being done. This layer should then also have it’s own reactive variable to detect changes, and to report this for replay/historical debugging. So, for every step, the total state would be the result of all collections and all data in the minimongo instance. The only thing I’d be worrying about here is performance. In a traditional Flux-like implementation, these are already in the state object and it can just be dumped. In the ‘minimongo as state’ scenario we would either have to query the minimongo instance for the data, or hold a copy of the data. Holding a copy means duplication of potentially all data in the application - obviously not such a great solution. Unless there is a way to access and return the minimongo collections as they aqe stored as references…

In general, I like this. I am fine with the differential update (instead of passing the whole tree), but the Redux purists would probably not like it. Thanks for the feedback.