Hey all! I have a 3 year old app that I run with 50-100 users. I have no major issues with the app, however when I first started it, it was very slow with certain pages where it needed to pull lots of data from the server, and then in some cases aggregate/calculate on this data further on the client.
To fix, I created a scheduled process that runs once an hour that does all the aggregating/calculating and stores it in a collection. This process first calls an external API via a GET request, and then based on the data it receives processes data in 0-5 collections, and then aggregates/calculates these collections into one aggregate collection. I then use the data from this collection in my frontend, which drastically cuts the load time down and is more than suitable.
My issue now is two-fold: 1) When this process runs, the front-end becomes unresponsive and shows a loading screen while the process is going on (since the published data is being refreshed). It typically only takes between 1-3 minutes, though it can take longer in rare cases. 2) Users are now asking for the data to be more real-time, meaning I need to run this process more frequently or at the very least allow it to be manually invoked.
I had on my TODO list to make the process not hang the frontend, but it kept getting pushed since it only occurs at most once an hour and is relatively quick. Now, it is at the top of my list as I cannot make it happen more frequently without causing potentially major issues.
My question is this: Assuming the process that runs is as efficient as it can be (it may not be, just bear with me) what are some better ways to accomplish this setup? Some thoughts I have include a way to “turn off” data publishing while the process runs so at least the loading screen does not pop up while it is running. Another possibility would be to attach a listener to the Mongo collections for changes and then only run the process that affects that collection on change. Perhaps I should turn off the real time data publishing but would then still need to not lock up the server and also somehow at least show that there is new data available.
I’m sure actual code would help more, but at this point I’m primarily focused on general tips and techniques I can look further into to improve the current situation. If no one has experienced this or has helpful thoughts, I can share more details, however being that the app is relatively large I’m also not totally sure the best pieces to share to help with this. Thanks!