If you’re comfortable with building restful endpoints from the scratch, surely that saves you a few iterations.
You might wanna consider using the latest version of Express, instead of the default Connect lib which is stuck a major version behind.
1 Like
I do see a value in being able to expose the same “service” (publication or method) via different transport mechanisms. That makes it convenient to compare the performance of both approaches to do some capacity planning in situations like the OP’s, especially if your first go to implementation is DDP. Another use would be to integrate a non-meteor app, without having to use a DDP client; something I’m doing in my own project
How can I use it in meteor? I have built node apps with Express before, but when it comes to meteor I just enjoyed everything being integrated.
@gothicmage, thanks for sharing your experience. I think I’ll go down a similar route – except that I can pay for expensive servers for the few hours there will really be the high traffic.
Definitely plannning a “high load” flag, and thinking about redux (I am not sure yet whether it’s going to be worth it. On the long run it probably will, also for the ability to use redux-persist to save traffic once more).
What to you mean with manual application of stalling ?
I didn’t know Grapher yet. It seems to be a neat mix of graphql-style queries on the existing infrastructure. Got to look into and think about it.
To scale to 10k concurrent users you’re going to have to run a lot of servers.
Some suggestions:
- Don’t use publications. Use GraphQL or REST as mentioned above. If you need real time functionality you could use publications or graphql subscriptions or something else. Meteor publications is where the pain point for scaling is going to be for you and most apps don’t actually need to be real time.
- Disconnect idle users
- Run stress tests before hand.
- I’m not a fan of Galaxy, but since it’s a short event, get set up on Galaxy and ask them to raise your limit from 20 containers to 40+. Running 40 * 1GB containers you may still run into trouble with a Meteor app and 10k active users. Depends what your app does though. (And again this is only if you need publications. With no need for publications/observers you should be able to scale fairly easily).
- Try redis-oplog.
3 Likes
What does going down a similar route mean to you?
What does this mean exactly?
YEAH, what does manual application stalling entail?
Grapher, I think it just speeds up the MongoDB bottle neck, you’ll still face a bottle neck at the Publish.
Great advice. Also, REST should be an option for you @retani, and NOT a stupid option, but a smart one. Also, I’ve seen code where Publications are NOT reactive – may be something to consider.
Interesting, how do YOU run stress test; I haven’t seen enough info on this around here.
Also, I’m truly curious how people are scaling outside of DDP or REST using Meteor, is there a hybrid DDP/Messaging Queue (RabbitMQ) or something similar that scales?
I only means that the progression of thoughts that @gothicmage describes - being faced with similar requirements - seem familiar to me. For example, I also came to examine Apollo but then decided that it’s not a thing for me. I am also considering redux now for better control over the data flow. I am also planning a “high load flag”
For me, it means integrating mechanisms for spontaneous reduction of API calls at the price of a less good user experience, but for saving the operability of the service as a whole. For example, we might use polling at one point. When I detect an overload of the service, I would set a parameter, the so called “hight load flag” on the server, which would then lead to the clients lowering their polling rate.
What is the “Pub link”?
Good point!
2 Likes
Not sure if I recommend using fine-rest.
As far as I checked, the old commits have been deleted, and the only changes are in naming and README. The author of the package has done that with few other deprecated packages and as far as my interactions went, it was very disrespectful for even simple questions or any attempt to collaborate.
Although not maintained anymore, simple-rest still very reliable.
PS: I’m not trying to insult anyone, just to prevent headaches if you choose to use the package.
PS2: redis-oplog does not work with namedContexts of SimpleSchema, if you use it, there is a open pull request to fix it.
I spoke with the author of simple:rest, and he is not going to add features or accept any of the many open PRs – it’s abandoned. The author of simple:rest is completely aware of what I did with fine-rest and has been helping me with the readme and other tips – he may even link to fine-rest from his repo at some point.
If you read my posts above, I explain and was complete open with what I’ve done to fine-rest so far. I’m in the process of adding token login instead of user name and password to fine-rest now. I’ll have that feature checked in soon. I also created a new example repo, you can get to from the fine-rest readme, that will show this and many other features of fine-rest. Beyond that, I’m not sure how much more I need to do to it at this point, but I will be using it for HA production enviornments that all my clients use.
I’ve already been collaborating on fine-rest with meteoruniversity and others in the Meteor community as well.
Importantly, I removed the main simple:rest package and only using the underlying JsonRoutes tech, so most-all comments were not valid and would not have made sense anymore – the project is moving in a different direction.
I’m open to collaboration on fine-rest just like I’ve been collaborating on the mup-classic project (just accepted a new PR), and if you’d like to become a collaborator, just PM me or otherwise just say so. Also, I’m already trying to help others in the fine-rest repo as well – just as I’ve done in the other repos I maintain.
If everyone thinks fine-rest should somehow not exist, I’m fine with pulling it into a private repo for my personal development – I just thought maybe others would be interested in what improvements I’ve made, and will make in this space.
Also, if I might add, I’ve been in this community almost from the beginning and always around this forum and other Meteor related communities – I am always willing to talk and discuss things.
Update: I posted an update on fine-rest below.
3 Likes
I appreciate the clearification. I got convienced by @raphaelarias and a superficial comparison on my side to better use simple-rest then. But now I think that fine-rest makes sense, too. However, @aadams I would suggest to make a note in the README.md to explain what you explained here (what the original packet was, what changed, why commit history was deleted). Otherwise, it does look suspicious (for example, number of contributors is also a metric for quality). Keep up the spirit of collaboration, cause otherwise, meteor might die with it’s community (It seems like a tough time for meteor, so packets are abandoned these days).
Maybe. But fine-rest does mention the original packages here:
Importantly, simple:rest is not apart of fine-rest, just the underlying JsonRoutes related tech.
Again, If you guys think it should be pulled, I’ll make it private to save everyone the headache. 
Indeed, but even the best repos with many contributors at the least have Issues – and if they don’t get fixed because it’s abandoned, then it doesn’t matter. I’d rather see a repo with fewer collaborators, where people are getting their PRs accepted and Issues responded to, than an abandoned repo with many collaborators.
And again, I’m open to adding anyone that wants to contribute to fine-rest as a collaborator – just PM me. 
I updated fine-rest to 2.0 today, and added the ability to log in with a simple, temporary, one-time token (not just user name and password as before), and added multi-database options for login too. Also, I’m getting closer now to removing the Accounts package dependency all together. I also updated the example repo (also found with a link inside the fine-rest readme) to take advantage of these options and much more.
1 Like
What advantages do you see in using current Express over the outdated Connect? Is it faster, maybe more reliable?
I kind of see an advantage in the separation and having total control over the middleware.
1 Like
I haven’t really compared the two, but the latest stable version is likely to be more reliable and performant than a version that hasn’t been updated for more than 2 years 
Note that Express had Connect as a direct dependency until version 3. I won’t be surprised if their latest versions still share many low level packages.
1 Like
I agree with this sentiment. I ended up just using Express for the Web API, instead of using JsonRoutes (fine-rest) – I felt like I was reinventing the wheel with fine-rest – I recommend doing the same. Express is extremely easy to use.
Apollo subscription seem more scalable, although I have no idea if that’s true as I haven’t implemented them. But from the looks, they aren’t making a bunch of observables?
I saw this thread and have a similar issue with my app. I’ve found the issue isn’t how many concurrent connections can one small Meteor container handle (BTW it’s a lot - like 400+ before I stopped testing), what you’ll run into before that is how many simultaneous logins (bursts of users) the server can handle before it slows to a crawl. And the answer is not much… like less than fifty for a small container.
Concurrent users just clicking around your app aren’t intensive on the server. Users downloading your app payload for the first time, all at the same time is highly intensive on your server.
I wrote a very detailed thread about it here - hope it helps:
50 is a number that I also found with load testing. Is there something wrong with your link?
I’ve replied to you regarding this: [SOLVED] Poor Galaxy Meteor Performance Serving Small Bursts of Users Load Test - #8 by raphaelarias
1 Like