Future of Pub/Sub and DDP

core
24

It is very difficult to say where the slowness comes from. I wish I could say. I mean “slowness to render when you click on something”. Sometimes more than 2 seconds of seeing “loading …” when clicking something.
We had maxed out CPU issues for a long time until I moved to bigger machines on EC2.
In these situations the app became completely unresponsive. Now it is just “not snappy”

I currently have two meteor processes running on an EC2 t2.medium using mupx. They run on different ports and nginx “ip_hash” load balances on a sticky ip basis.

Mongo runs on a separate t2.small and never shows any load on the graphs on the EC2 console.
As mentioned, we do not have oplog tailing enabled.

We have separate node processes that write to / update the same collections so contention /locking issues here is not out of the question. On the other hand, simple admin pages reading from small collections also aren’t ever very snappy.

Is there a way of gaining insight into how much data is being transferred via ddp between the server and a client?

I’ve paid for the basic Kadira but this hasn’t given me any clues, nothing looks obviously wrong.
Server CPU profiles just showed a lot of time on “other” or something, and profiling of the app when it was flailing at 100%cpu never succeeded. Nothing helpful from client-side profiles either. From memory the bulk of time was within Meteor “processing Mongo documents”

Number of users seems more or less proportional to CPU usage. We have never even had number of users > 15. Users see a lot of graphs that can be resized etc so there are some unavoidably heavy subscriptions.

I’m using most of the suggested kadira packages except for kadirahq/subs-manager
This made things snappy but at the expense of up to date information.
If Pub/Sub/ddp got things totally right why would there be any need for subs-manager?

We have been running on Meteor 1.1 for a long time.

I have a 1.3.4 version of the app being tested in Staging. It doesn’t seem obviously faster.

I wrote the initial sketch of an app then it was fleshed out by a relatively junior developer.
There is definitely plenty of sub-optimal code in there, but I would like to be able to work out what parts to investigate from more of a “distance” first.

Any advice on diagnosing the slowness please?

METEOR@1.3.4

standard-app-packages
iron:router
accounts-password
multiply:iron-router-progress
ian:bootstrap-3
ian:accounts-ui-bootstrap-3
ephemer:reactive-datatables
wylio:mandrill
comerc:bs-datetimepicker
kadira:debug
meteorhacks:kadira-profiler
meteorhacks:fast-render
standard-minifier-css
standard-minifier-js
email
meteorhacks:ssr

.A

25

So first, a disclaimer, I am a full-stack developer, which means I have 80-85% of what an expert developer in their respective discipline has. So I’ll put on the had of a db admin on as my hunch is that this is where the problem is.

My experience is that db engines don’t always push the CPU through the roof when they are overloaded, they push ‘other’ processes and OS ones. MongoDB, even great as it is, requires frequent data to be in-memory (this is why they like SSD so much). They also recommend to remove from the OS last access time monitoring as it adds overhead.

So you need to start by diagnosing whether you have cache and index misses. There are a few tools out there to handle this. We use icinga2 as our monitoring tool, we set it and forget it. Now, you may need to probe deeper to see what is going on.

MongoDB often needs a lot of VM’s to run well by design (they claim it’s for redundancy). So it seems to be a built-in ‘feature’ that you need more machines and memory to get the job done :slight_smile:

1 Like
26

Thanks Ramez, I’ll take a look at the Mongo side of things and let you known if I find anything or change any behaviour radically by changing things there. I may as well set up a replica set and two slaves. Would you expect that to change anything? I may as well try switching to oplog tailing?

27

Refer to this previous post of mine about using Meteor with MySQL

Yes, we successfully updated our app to Meteor 1.4, which required only minor changes to our code, and are now enjoying lower CPU usage mostly thanks to the Node.js v4 upgrade, which included a newer version of the V8 Javascript engine.

We can hardly wait to see what Node.js v6 will do for our Meteor app.

I would like to ask the 50 million dollar question:

Who chose to elevate Facebook to such supreme heights of holiness? Why are we expected to drink the poisoned kool-aid and accept the Facebook platform and libraries to represent the perfect vision of how our web apps should be designed or implemented?

Our app is not a Facebook clone or Social Networking tool and neither are most peoples’.

The Internet has been around for a long time and is used for many other things apart from social networking!

Sure I admire what Zuckerberg and Co have done, but that doesn’t mean I want to be exactly like them and do what they do!

To me, Meteor resembles Coca Cola. The company succumbed to unfounded FUD about their original formula and replaced it with New Coke, only for customers to realise that new was not necessarily better and that Coca Cola Classic was right formula all along - just like Meteor Classic.

MDG - are you listening?

2 Likes
28

I would not characterise oplog or binary trailing as a design fault - it is a valid technique for implementing reactivity, but it will reach a performance limit.

The rate of reactivity is limited by the speed at which Meteor can read the oplog/binary log, execute reactive code and push relevant updates to the client.

As long as this limit is greater than what is needed for your app, then there is no problem.

As you know, I’m using MySQL instead of MongoDB with Meteor as I find it performs better for our app. I have been continually finding ways to optimise our design and implementation to raise the performance limit higher and higher for a single server to minimise the size of our cluster.

29

We only looked at Meteor because it had pub/sub, i.e. realtime sync that allowed for very complex business logic. (Yep, it’s B2B.)

We liked Node because that reduced the number of languages, and Meteor checked that box.

We figured Mongo, or something similar, because we need a flexible record/doc structure for a few tables/collections — and Meteor checked that box.

We needed a build process, and Meteor checked that box as well. (And 1.3 has gone a long way toward cleaning up Meteor’s failings there.)

We needed users and some form of self-managed accounts, and Meteor helped out there too. (The profile subdoc is annoying, but livable.)

I could do without Blaze, or React for that matter (which I consider a step backward as it seems the JS alternative to feeding PHP to the browser). We were porting a large Knockout codeset and, for better or for worse, I find the Handlebars syntax bloated and I’m not a big fan of how it manages the DOM thru strings, so we only use it to bootstrap the page(s). At some point, we’ll factor it out.

Meteor has its warts and its weaknesses, but pub/sub and DDP certainly aren’t among them. I get that MDG is integrating Apollo in order to expand the architectures it can support, but I like how light DDP is and how I can write many of the business rules once, attach them to a transform (model) on the collection and use them on both server and client. I look forward to a long and healthy life for pub/sub and DDP :slight_smile:

8 Likes
30

@vlasky,
Thanks for your comment.

The reason I believe it’s a design fault is as follows:

Imagine your app scales up quite a bit. With that scaling, oplog is going to scale up as well.

So each new user will result in increase in oplog changes. This will slow down ALL your instances of meteor. So you are no longer scale up by adding new instances for new users only, you also scale up by adding more instances as your current instances are getting overloaded! I can’t find an excuse for such a lousy implementation to be honest. It’s small-time reactivity.

In fact, mathematically, there could be a point where you can’t scale up any more, and any new user will result in automatic degradation regardless of number of instances.

Now, concerning your mysql solution, it is better than Mongo as we had discussed, as are postponing the breaking point:

  1. You set the update interval manually – which you can increase to reduce CPU hits, but at some point the app is no longer reactive real-time
  2. You set more complex rules to filter the binlog – then again, those rules have to be executed for EACH subscription, for EACH binlog change, which will eventually hit the limit too.

Solution: The DB itself has to be a realtime DB that is designed for reactivity. The only popular / robust one we can find is rethinkDB. We are working on the Mongo replacement driver as we speak. But all these client-based solutions will suffer at some point regardless of how you implement it, as you are (1) using a scripting language – JS, (2) parsing a lot of data to find what is relevant for you.

2 Likes
31

What a smart way to put it…

I’m curious, have you guys evaluated Feathers.js for scalability? It supports RethinkDB of course. Here’s a video with @joshowens where he goes into some of the technicals with the founders. It seems Feathers.js is where MDG is pivoting, in part, possibly due to scalability options this new style opens up.

32

Please explain this more :smiley:

I am not sure I agree with that at all. If anything, the pivot is to Apollo and will likely be a sole focus at some point in the future.

1 Like
33

In a sentence: from your interview I gathered Feathers.js is a lightweight opinionated framework and api around stable best of bread npm packages that seem to check all the boxes an application needs to take it from zero to “go” in a relatively short amount of cycles.

34

While I would agree that those were early goals of MDG with Meteor, it seems the pushing back of releasing an NPM released version of Meteor until later puts that in jeopardy, in my mind. No one has come back off the Apollo project once they start working on it, my fear is that @benjamn might keeping working on Apollo stuff more than Meteor stuff once 1.5 is ready…

3 Likes
35

But @joshowens, I thought the “new pivot” by MDG away from Meteor was all about embracing the NPM ecosystem, opening up new DB options via Apollo, and making a concerted effort to target the enterprise in part by adopting FB-style-tech.

In all these aspects at least, Feathers.js is already there.

I read somewhere the thinking in MDG is, why NPM-ize Meteor fully when no one outside of the Meteor community would use it? So in but not out.

It seems on the surface at least, that Meteor-classic is “done” and all further efforts by MDG will be in the Apollo cost center.

I guess when I suggest MDG is pivoting to a Feathers.js style strategy, what I’m really suggesting is they’re pivoting to Apollo (and away from Meteor), and the new Apollo stack is more inline with where Feathers.js is already at.

36

The future of MDG is data-related. Disinvestment in Meteor will start. People misread the Blaze announcement, and started a UI war.

What it really means is that third-party frameworks will be leveraged as much as possible when they are not core (e.g. UI). And right now, only data interfacing is core.

We are almost done moving to rethinkdb (will share GitHub shortly) and we expect in the future that the community to play a (very) leading role, as we started doing. Hopefully without having to fork.

1 Like
37

But when you considered Apollo instead, what conclusions did you-all come to?

Indeed as @joshowens alluded to above, this has already begun with 1.5.

I think you’re right to a degree, although I pointed it out early that when MDG embraced React, they’ll eventually embrace the entire FB stack that goes along with it (which was my real worry). Yet at the time I knew nothing of FB’s GraphQL or MDG’s spin on it (Apollo).

Exactly, and this is where Feathers.js is already at. They’re building out APIs for supported DBs and authentication mechanisms. Wrapping a lightweight layer around middleware such as express, socketio, etc.

38

We can’t wait for Apollo, we have a deployment that is choking because of the oplog mis-deign.

Also, we don’t really want to invest in FB ecosystem, our stuff works and works well, so why bother switching just because something is in the best interest of the platform provider? We like pub / sub.

Finally, we have resigned to the fact that we have to start putting devs into meteor anyway to get what we want.

39

But they’ve already shown they aren’t super interested in things like making a build tool (see Saturn as an example of an effort they shutdown). They are hyper focused on the data loading pieces. Honestly, I don’t blame them for the choices, as a business it makes sense to me. Just sucks for the community that it will continue to happen slowly and quietly, as far as most are concerned.

3 Likes
40

I understand, but aren’t these two in conflict? The Apollo stack revolves around and integrates with FB-tech. Embrace the backend without the front? Won’t you miss out on integration points?

41

Please elaborate on this. Are you saying you think they’ll build out Apollo proper, without supporting infrastructure, in the classical Meteor sense.

42

So the work you-all are doing to ingrate RethinkDB into Meteor-classic won’t be lost by moving it behind Apollo?

43

I personally think that Meteor is changing forever, in a direction that supports new business, not current community. So regardless, we are on our own. So might as well start digging into it and improving it and start sharing with the community.

If things go south, we will simply maintain our meteor fork and slowly migrate out into expressjs, socket.io and so on. The Meteor code base is very solid and can be a great foundation.

2 Likes