Meteor Scaling - Fusion Mode [Status: Beta]

[diaconutheodor]

Update. This idea transformed into a package:

cult-of-coders/fusion

How it started:

Hello guys, now that we’ve solved the scaling of Reactivity (RedisOplog), ability to have a relational database inside Meteor (Grapher), provide a learning curve for juniors and give the masters in Meteor some amazing ideas for enterprise apps (MeteorTuts) it’s time to solve other problems:

1. What if I want a Meteor app where I don’t want websocket connection? But what if I want websocket connection at some point in some parts of the app.
2. I want Meteor pages to load in an instant…

Conditional Websocket

Now, websockets are expensive, and depending on the app, you may not need them. Especially if you are building a SHOP or a presentation PAGE for your business, you don’t really need reactivity, but in the back, you may need to do some api calls, expose some methods for contact, get some available jobs, read from the database, etc.

Blend CSR and SSR

The idea is that when the page initially loads, it should render server-side(everytime), send it to the user, in the background load the Meteor scripts, and then it becomes a one-page app. If the user clicks fast enough on another page and meteor scripts aren’t loaded, that page will, again, load in SSR fashion.

This can be done via HTTP and not DDP. Therefore, we can emulate a DDP Driver for HTTP for receiving requests, and translating them into Meteor.calls.

Meteor.call('jobs.find', function () { .. }) // Does a HTTP Request
DDP.engage(options)
Meteor.call('jobs.find', function () { .. }) // Works via DDP

So we need to find a good way to work with Accounts, Security.
When a subscriptions starts, we automatically need to engage in a connection.
We need to make this as seamless as possible for the user, fully backwards compatible and keep the Meteor way “simple, magical, hackable”

The fusion of SSR and CSR part may be harder than we can even anticipate, but I believe it will complete the web.

---

What we gain by doing this ?

• We minimize socket connections to the server (Better performance)
• Ability to have Pages that load absolutely instant, and have great SEO
• Ability to have apps without websockets at all (or sockjs for that part)

Before I dive into this, I need to know your opinions. I’m always open to the ideas of the community. Let’s explore some pitfalls.

[alawi]

I think this is a great discussion on another part of Meteor that has potential for improvement.

As a starter I think we need to outline exactly what existing pain points are we trying to address. I know other frameworks (Next.js) has been pushing hard on all SSR pattern, however from a user perspective, I really don’t see much value in that. The biggest issue with SPAs of the past is the initial load time is linearly promotional to the bundle size, and this has been improved a lot with introduction of dynamic imports. So in my mind, going all SSR is the other unhealthy opinionated extreme of the spectrum with little value added.

I think the focus should be in maturing the APIs and libraries around Meteor’s SSR which is a relatively recent addition to the Meteor core toolkit. I’ve used a dynamic imports and SSR to reduce the load time for the an app from 6s to less then 2s. However, I had to introduce some workarounds and third party libraries to pull the user data, I’ve captured the limitations here and I’m happy to see work being done recently by MDG to improve the API.

In addition to improving the SSR APIs, I think we need to have control over where the merged CSS file is being injected by Meteor, since the injected CSS will impact the critical paint path. This was delaying my load time by 590ms and I’ve captured the issue here.

The browser also spend a lot of time executing the loaded JS files. The larger the JS file, the longer it usually takes, so any reduction is the core Meteor bundle is also welcomed. I’ve also noticed that the more split points I introduce by using dynamic imports, the larger the Dynamic-version.js file gets, in fact this file grew to almost 6% of my bundle size, you can see the chart here, so I think this could also be improved.

With regards to conditionally engaging DDP and having RPC of HTTP, I think that would be awesome! and as you said it’ll provide advanced developers with full control over the data transport layer.

Overall, my personal opinion as a user, is that Meteor is on the right track with SSR and the community can introduce some libraries to complement the API’s and enhance the user experience in the near future. However I’d wait a bit until the recent SSR improvement work is complete. Also, I’d love to see RedisOplog getting more testing and marketing. Concurrent users scaling has been the uncertain part of Meteor, and the fact that it was solved by community (thanks to you) is really exciting!

[diaconutheodor]

Thanks for your response. Only SSR pattern is taking a leap backwards for sure. And according to Google, 53% of users drop if the page loads in more than 3s. This means that business people will prefer SSR over CSR.

I hate doing many things at once, so I can only focus on the conditional websocket part. I’m very glad there’s focus on the SSR part, I think I’ll leave MDG to this, they know best, and hopefully by 1.7 we’ll have a solid implementation.

In regards to your app, I believe 2s is still a lot. Especially 590ms for css only. Maybe the future is inline-css

In regards to RedisOplog, it will get more testing and marketing, I’m already in discussions with other companies to implement it in their no-longer-scalable-because-of-oplog apps. We’ll extract some nice case studies, and Meteor can brag with it for sure. It’s a very exciting stepping stone, but there’s still a lot to do.

[alawi]

@diaconutheodor

I can only focus on the conditional websocket part

Make sense, to get that going, I’d assume we’ll need to change Meteor core DDP package?

@diaconutheodor

Especially 590ms for css only. Maybe the future is inline-css

Yes, it can still be improved, I was able to get around 800ms on my local machine on fresh Meteor react project. However, as stated, as far as I know, we currently don’t have control over the merged CSS injected by Meteor (those you in-line in the JS files). I’ve moved the imported public CSS out of the critical path, and in-lined all the CSS I need for the server rendered login page, but any CSS I wrote in the code gets merged and injected at the head, and I can’t instruct Meteor to delay that.
https://forums.meteor.com/uploads/default/optimized/2X/2/2c18eb8b91441687b8c5a19ca5cf46dc2e8232b7_1_690x349.png

The first interactive paint is what I think you’re aiming to improve, and DDP/Sockets initialization seems to be slowing that.

[diaconutheodor]

Make sense, to get that going, I’d assume we’ll need to change Meteor core DDP package?

Truth. I posted a question here: https://github.com/meteor/meteor/issues/9349#issuecomment-343724758 as I need some input from the guys that coded ddp-client (Looking at you @sashko)

The first interactive paint is what I think you’re aiming to improve, and DDP/Sockets initialization seems to be slowing that.

Indeed, this will also improve server performance and ability to handle more users that view the page at the same time. My problem is why establish an websocket if you don’t really need it ? This makes Meteor suited only for some type of apps. What about presentation pages, shops, many others ? Why not have the ability to become reactive only on some special pages ?

[alawi]

My problem is why establish an websocket if you don’t really need it ? This makes Meteor suited only for some type of apps. What about presentation pages, shops, many others ? Why not have the ability to become reactive only on some special pages

I’m sold on conditional sockets! makes total sense. Also I think MDG might face similar decision/DDP refactoring as soon as they plan tighter integration of Meteor with Apollo. If I’m not mistaken, I think Apollo by default runs on HTTP and only leverage web sockets when subscriptions are defined on a page.

[diaconutheodor]

I’ve been toying around with it for a bit and I made a bit of progress. Managed to integrate it with Accounts Password, Meteor calls now work with this.userId. So I managed to hack the authorization layer via the HTTP.

The problem I’m now tackling is the automation of engaging in a DDP connection once the first .subscribe() is done. However, there is a big problem, because Accounts automatically subscribes to ‘meteor_loginAccountsConfiguration’ by default, so basically it would mean that if you have Accounts, the first subscription automatically engages the DDP, so it’s useless.

I think it would be best to manually do this:
DDP.engage()
DDP.disengage()

And let the developer full control over this. For example, you can do things like (if you’re on the page for a long time without any activity, just cut the ddp loose, and resume it when you come back)

[diaconutheodor]

BOOM!

cult-of-coders/fusion

Conditional websockets in Meteor are now a reality. I’m curious how many bugs you find, it’s currently a POC, but it should work just fine in 99% cases. Read the README. (It works with Accounts)

[diaconutheodor]

@diaconutheodor

The idea is that when the page initially loads, it should render server-side(everytime), send it to the user, in the background load the Meteor scripts, and then it becomes a one-page app. If the user clicks fast enough on another page and meteor scripts aren’t loaded, that page will, again, load in SSR fashion.

This already works so nice with “server-render”. Basically you can use react-router 4, with a BrowserRouter for client and StaticRouter for server, and pass in the same routes! It’s the first time I used it and I’m amazed (need to add static-html and remove blaze-html-templates) for it to work.

And if for example you are subscribing or using createContainer() || withTracker(), you can fetch data directly server-side and pass it to the components, no change required! But you have to be careful to only fetch what’s needed. And also maybe do a Meteor.subscribe = function () {}; a dummy function on the server.

I am in awe, Meteor is now so performant I can’t believe it. We have ability to work without websockets but still benefit of authentication and meteor calls, we have server-side rendering that blends so nicely with React. We can now reach very large scalability, especially if we put .css and .js into a CDN.

And you can very easily cache the SSRs so it’s way way speedier:

Server-render and SEO

I can show you how we use it in our application: lib/main.js import React from 'react'; import {App} from "../imports/ui/app"; import {Helmet} from 'react-helmet'; if(Meteor.isClient) { import { render } from 'react-dom'; render(<App />, document.getElementById('app')); } else { import { renderToString } from "react-dom/server"; import { onPageLoad } from "meteor/server-render"; import NodeCache from 'node-cache'; const renderCache = new NodeCache({stdTTL: 432000}…

Absolutely beautiful.

[alawi]

Wow, great work as usual @diaconutheodor ! I’ve being using React Router V3 and manually handling server side routes, so this setup is definitely an the right way forward.

React is becoming a standard in front-end development and it’s growth is not slowing down, so first-class support of React by Meteor will definitely increase the framework adoption. In Meteor’s roadmap they’ve a section for “Out of the box support for advanced React features”. If we’ve something like meteor create react-app that would generate a React skeleton with SSR, React Router V4 with dynamically imported components at the route and conditional DDP then I think it’d be a great head start for any app. If you’ve a blog or static page without reactivity, you can just hit that command, deploy and be sure that your page will load instantaneously and it would scale as any node server (no tailing or the extra overhead of socket management at the server). As the site/app grows in features, reactivity can be added on demand using DDP engage and Redis-Oplog. I think all the solutions for react are here, they just need to be packaged and presented the Meteor way to the user and it’d be better then Next.js for React since it’s more flexible.

I’m really excited about this, I’ll give this a spin with a side project to see how it behaves.

[diaconutheodor]

Regarding meteor create react-app

Since I already had it for testing purposes… why not?

cult-of-coders/meteor-ssr-react-router-demo

[alawi]

Thank you!! I knew you had something for testing, you read between the lines, I just didn’t want to ask for it directly

But I think we need to have something below, right now it has only simple-todos-react.

[diaconutheodor]

We should have a sort of aggregated list of meteor boilerplates somewhere.

And sorted by the number of stars, and completely searchable.
If you want to add a new boilerplate, just add a GitHub Repo link.
Meteor will interogate the github api, sync stars daily. And comments (via disqus) enabled at each repository.

And ofcourse it’s going to use the “fusion” package because we don’t really need an WS opened

Any thoughts? I think this would be extremely useful, instead of just adding a boilerplate to react app, some people may want a react+redux+redux-router, others want a react+mobx+XXX

I think this will be useful for people starting new projects and old-school Meteor devs (like me) to try out cool new stuff.

[vooteles]

I think the idea is great (I’m currently in the process of moving my project to Vue and having others’ setups to look at is of much help), but it would probably be easier to make this happen without the direct involvement of MDG - i.e. not have this list shown in the shell with meteor create and instead just have a simple website available with links to repos with the clone command right in the middle. I’d say git clone is not much more trouble than meteor create in the shell.

Having the boilerplates tied directly to meteor create would mean that MDG probably would need to verify the code to ensure that it meets a certain standard (both with regard to its quality and absence of malicious code), as users would assume that such example code is approved by MDG and safe to use. This would be burdensome on MDG and even if MDG would have the resources to do that (which I doubt) the whole process would be too slow for the list to gain momentum. So I think this is a perfect opportunity for the community to step up and make something happen, and help MDG out in the process.

[alawi]

I think this is great idea as well. So basically something like NPM/Atmosphere but instead of packages it’s for Meteor boilerplates.

Do other modern frameworks have something like that? does anything similar exist?

[diaconutheodor]

@vooteles
You understood correctly my idea.

@alawi
I haven’t seen, I searched a bit but no ciggar.

This can be a great opportunity to test fusion and ssr

[alawi]

Next.js maintain a list of example repos on Github, but as @vooteles stated this approach might end up with bottleneck in the maintenance and growth of the boilerplates list in addition to draining of resources from the core dev team, so I’m completely agree, a standalone community powered platform makes a lot of sense.

This can be a great opportunity to test fusion and ssr

Agreed! an instantly loaded SSRed/Fusioned meteor app with all the active meteor boilerplates, sounds like another great addition to the Meteor community!

[alawi]

Here is the closest thing I found for react that implements some of those requirements, since the data in that tool is maintained centrally, you can see many requests to add boilerplates in the comments section, so it could definitely be improved and made into a standalone platform.

Another approach I found here in which they’re using a GitHub to maintain a list of boilerplates involving mobx, again this is not ideal since it’s centrally managed without any search capabilities.

And some developers just blog (example of a blog made a month ago) about the latest state of the boilerplates ecosystem which is obviously less then ideal.

This seems like a gap in the entire ecosystem frankly and there is opportunity for Meteor to lead the way - yet again

[jamiter]

A little of topic, but I wanted to mention it’s already possible to use DDP for Apollo. For queries and subscriptions:

Swydo/ddp-apollo

To be honest, there is no reason to expect a much tighter integration from MDG. But that’s for another topic.

[tomachi]

Great idea. Render it server-side everytime. If the user clicks real fast, serve up the next one again from the server. Later the app takes over. Nice.