I am starting to write an application which requires stock price changes to be sent to 1,000 clients or so. I have already written a prototype using the Meteor pub/sub. Considering the volume of data and number of clients, what is a better way to write this application to be able to scale ?
I have seen several add on packages which claim they do provide a direct Socket IO solution. But many of have not been touched in ages.
I like to hear suggestions from people who are intimate with Meteor and its roadmap.
This is not directly related to Meteor, but you might consider a platform like Apache Kafka:
Kafka™ is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
Martineboh: Have you actually deployed your app with large scale data updates ?
For my prototype I have deployed my app to meteorapp.com and use Mlab for hosting the mongo. I am seeing some skips in data updates and delivery to the clients. Not sure if this is due to network latency between the DB and app server or from app server to clients. I am about to use kadira to get more detail. Thanks for your comment.
Tom
@tomvolek You could be having delay updates for several reasons. Network latency is a major culprit when your mLab DB is not hosted in the same region as your app. With the right optimizations, using redis for pub/sub with redis-oplog instead of mongo will drastically improve your app for scalability. You can show some codes for further help…