Why just don’t rewrite some meteor’s parts in native langs (c++, golang or whatever)? Say, replace socket server it would process it’s heavy parts easily, the same for ssl, the same for pub/sub. The typical user even will not have to change his code but server becomes not multithreaded in its original sense, but at least less resources hangry and easily horizontal scaling… Why all of you just want to make it running in javascript only, in one event loop? After all V8, the core of node, is written in c++.
3 Likes
I don’t take your word for it that the websocket package used in Meteor is indeed poorly performing.
All we have from you so far is a claim that you failed to substantiate. Your VPS environment is not suitable for any performance testing anyway. You didn’t provide us with any measurement data (let alone reliable one), nor did you show any piece of code that makes your story plausible or believable, such as that you have “set up a separate socket server written in Go” and “written my own pub/sub to replace the native meteor implementation”.
You claim to have already solved these problems, yet you don’t stop demanding that we, as a community provide a solution for you to the very problem that allegedly needs no solving anymore. Why all the fuzz if you have your own perfectly performing websocket server written in Go that you can use with Meteor?
2 Likes
Please check the PRs for Meteor. There are already some that are experimenting with ideas discussed here.
1 Like
We use Meteor v1.8.1; we didn’t update it to the latest version yet.
The app has 2M+ users. This is the default meteor usage; we haven’t used any special trick. We are using AWS with Horizontal scaling.
4 Likes
Your VPS environment is not suitable for any performance testing anyway
If I have 2 same apps running at the same VPS and the one is “not suitable” as you say for this “hardware” but the second - suitable, as I say, because it works fine, I would say that the first app is not suitable for using. All the differences between apps was just the pub/sub subsystem differ although having the same publications and subscriptions
You claim to have already solved these problems, yet you don’t stop demanding that we, as a community provide a solution for you to the very problem that allegedly needs no solving anymore. Why all the fuzz if you have your own perfectly performing websocket server written in Go that you can use with Meteor?
Ok. Lets finish at this point. You play with your toys and I’ll play with mine’s…
Nice. Could you give as the resources used for peaks of usage?
I am sorry, but your question isn’t clear to me
How many resources used by you application when 2M+ people come to your site and if in this situation it is automatically horizontal scaling (or manually? Or you have constant resources (VMs, CPUs, RAMs)?) - what are these resources?
1 Like
I didn’t say that a VPS was not suitable for your app. I wrote this: “Your VPS environment is not suitable for any performance testing anyway.”
It may very well be that there was a bug in your app back then, which you mistakenly attributed to one of Meteor’s component. Maybe there were circumstances when method calls or subscription code escalated in that app of yours due to bugs you never discovered, perhaps up to the point of endless loops on the server, and that caused 100% CPU usage. Then you rewrote a few parts, and the bug was gone. It wouldn’t be the first time in software engineering that a bug appears to be somewhere, whereas in reality it was caused by something else. I guess we will never know.
You can send me an email at sanjay @ deligence.com and then I’ll share as much information as I can. Can’t share all these details here publicly.
maybe it useful to share with us what apps you have running or have in mind to create?
Well… Currently I have 2 meteor apps running.
First is a quiz system for american’s young jewry. It has a few UI clients - administrators or operators but thousands of participants. System sends sms to a groups of young jewry with questions about their’s history via twilio or nexmo. They respond. Some wins, gets scores and so on. This is still in meteor 1.2.1. It works fine just because of few UI clients. But it had one issue right as I told before when I’ve created second separate simple meteor app that was necessary once for a raffle with a simple UI - system just randomly selected connected users (with a phone) when a button pressed and all of them gets message who wins through pub/sub. This raffle server was connected to first quiz system’s server for some simple authorization and getting user’s names by their phone numbers. It was a dead simple system of 2 subsystems. And and stucked when users became 50-100. So the raffle at that point mostly failed. The main quiz system works about 5 years and I sometimes makes some small fixes or additional workflow changes.
The second app is my personal blog. Firstly it was written in meteor, don’t remember exactly, by I guess it was version of <1.2. Later I was struggling with it when there was breaking changes in 1.3 and now it stopped at 1.8 version. It has a self made pub/sub as I mentioned before, not perfect, not as mature as meteor’s original (it is not very well in a bad connection or sometimes not reconnects when laptop sleeps and woke up and has less abilities) but it lets my server’s CPU almost sleep: 2-3% with about 30-50 users online and about 20-30% with about 400-500 concurrent users online. I mean server, not meteor’s server application! I have such a peaks when deploying new chapters of my books and users comes. Not sure it can be interesting to you, blog is in russian but you may take a look at it: https://yasinskii.ru/. In that blog besides by books I have some posts with all my troubles with meteor I faced but they in russian too and don’t think you will read it. So right now I plan some changes to a site, some other integrated tools for a writers, although I have some - I write texts right on site and readers may correct mistakes (there is some git-like subsystem for text changes for users) right when they read (many people likes this and feels as involved in process of book creation). But I plan to expand this tools. That’s why I was attracted by the changes in meteor state today.
1 Like
Nice, I like the book idea, I was looking for something of that nature for personal blogging and publishing as well.
For the raffle winning message, you’ll need a socket for sure, so stateless Rest won’t work. But I would use a DDP streamer and bypass the merge box to communicate the winning message. I’d have also used JAMStack with DDP backend for the client that expecting thousands of hits.
And for you the blog I would have used Meteor methods without pub/sub. I think those two approaches would have solved your issues 
And for you the blog I would have used Meteor methods without pub/sub. I think those two approaches would have solved your issues
Maybe. But I liked meteor for all its features. Without pub/sub its just like any other framework - Next.js, Nuxt.js, Sapper. With its features of course, but nothing special. Besides, for the last time I prefer concept of separation client server and api server. It lets more flexibility for tooling and client’s demands.
For example take a look at https://github.com/create-go-app/cli - you can choose any framework for client just like meteor (React, Vue, Svelte), api server is in Go but you can choose router framework, database, etc. Plus using websockets in Go is like a breath. Or just REST. Http, https, http/2, SSE, automatically get let’sencrypt sertificates. Plus green threads and normal threads. One interesting thing about Go. If app has several green-threads running in one OS thread and one green-thread “hangs” by this blocking other green-threads, Go core will run new OS thread and move these other non-hang green-threads there and they will continue to work. I was shocked when new this. For last year Golang is like meteor for me 5 years ago…
Maybe you will be interested in my project for collecting of books links (aka url aggregator). Nuxt.js, Golang server, Mongodb, SSE for simple reactivity, recommendations… https://blinks.authors.city/
Anyway, thanks for interesting conversation.
1 Like
Yes I understand where you coming from now, but that is what I was proposing, we have that setup as well with some apps using SimpleDDP (or you can use GraphQL) and Create React App but you can use any front-end framework.
The advantage here is that you have an optional real time, accounts, SSR, hosting, and rich ecosystem with JS end to end.
I have tried GoLang and I wasn’t sold on using it for web development, if I decide to branch out of Node I would pick rust or Deno instead but of course Go will get the job done (or any other back-end framework for that matter, Akka or Spring for the JVM or even .Net would have been good choices here).
I enjoyed the conversation as well, and thanks for sharing your work, yes I’m interested in your books aggregator, we can a chat on this on slack or another thread I have some ideas on this topic
if I decide to branch out of Node I would pick rust or Deno instead
Rust is a nightmare in development, ugly in syntax but perfect result. Deno… It’s just like nodejs but from other point of view, how could it be used in node branching?
we can a chat on this on slack or another thread I have some ideas on this topic
Unfortunately I don’t use slack (in company we use local mattermost, so I dropped using slack) but you can find me in skype - andrew.manson.noperapon or just write me an email to manson.andrew @gmail.com
Yeah, I don’t like Rust syntax either, but if I’m sacrificing the full-stack JS advantage, might as well get a perfect result. For Deno, it is the the same as Node (and that’s not a bad thing for me) in a sense that it runs on V8 but it is different enough in the API and architecture to worth exploring, at least for me, so I would give it more credits and keep a closer eye on it.
What do you mean by socket? Websockets? Since you mentioned you projects from long time ago are you using mongo oplog? Is it enabled in production? Can you share information of profiling you did to find bottleneck? It would also be nice if you could share the code that improved performance, maybe someone who works on meteor core would take a look at it.
What do you mean by socket? Websockets?
The same as Meteor means. That is a core of it.
Since you mentioned you projects from long time ago are you using mongo oplog?
A far as i remember, no special mongo switches was used. Just default mongo instance as it was after installation (there was some other projects running on it)
Can you share information of profiling you did to find bottleneck?
Never thought It would be useful for someone else so nothing stored. As Meteor rose from the ashes with a new team I’ve just interested in that I’ve stumbled here a time ago.
It would also be nice if you could share the code that improved performance, maybe someone who works on meteor core would take a look at it.
Don’t think this could be useful because I didn’t improve the existing core of Meteor, I’ve just rewritten it (to be more precise - application’s server logic, not meteor’s. I’ve just stopped using meteor’s engine and its DDP - switched it off) by using the side socket server and most logic. The meteor’s server side part (in javascript) that uses this separate socket-server is a simplified version of the similar logic the meteor’s team has implemented.