Hi Guys,
During my studies, I deal with Meteor and in this connection with the question:
What is the best (temporary) way to store large amounts of data / aggregated data?
Maybe someone has recommendations in this regard?
best regards
Kristina
Hi @Kristina, welcome to the forums!
There is no one correct answer to this. It depends on how much data, what needs to be done with it, how many users the app has, etc.
Can you give a bit more information?
Just to get started, if you’ve got enough RAM, the easiest is to store the temporary data in a variable, then it’s immediately available to the app, and once you clear the variable (either by going out of it’s scope or setting to undefined), the garbage collector will automatically remove it
2 Likes
Thank you for the effort!
The question is more general, so I have no specific information. The aim is to understand various options for storing large amounts of data and to recognize their advantages and disadvantages. :ich sehe nichts Böses:
So the option you mentioned helps me.
1 Like
Another option if you are constrained by RAM is to cache it to disk.
I usually use sqlite for this, as it’s very simple and removing the data is as easy as deleting a file.
If you need more powerful query performance, the data can be temporarily added to MongoDB (or another DB, but Mongo is the most Meteor relevant).
You will need to set the data to expire, using expireAfterSeconds or manually.
If querying is not needed, and you just need to work on a larger chunk of data without leaving it in memory, you can use streaming APIs to parse the data chunk by chunk (usually lines).
This means reading only the smallest logical unit at a time (reading from the network, a socket, or a file), and writing each resulting chunk to a temporary file as well.
Once all the data is processed into this temporary file, you can stream it back out to wherever (eg to a client who is downloading)
I’ve never needed any methods more complicated than these
1 Like