This is pretty easy when it’s not a lot of data. You just do a query and use papaparse. It seems like this strategy starts to fall apart when you’re querying several thousands of rows.
Is there a better strategy? I’ve seen some apps will have you hit “download” and they’ll send you the file (or a zip) as an email attachment (in a transactional/auto-generated email).
Has anyone done this? Did you use a lambda function or something? How do you not shit out and run out of memory?
I used exactly the strategy you mentioned.
I used a lambda function that queries the db, generates the file, uploads it to S3 and returns a url.
I didnt sent it as an attachment but used something similar to google drive, I put this transaction on a collection and when is complete I just download the file on the browser. If the user closes the browser he can later download it on the “downloads history” with a download button.
About the memory, what I did was to first check how much memory the lambda function is taking, usually our files have almost the same amount of data so its very consistent so we didnt have much issue with that.
Did you have to paginate the results or anything? Or can you just query 100k records and plop them in
I haven’t had to do this with mongo / meteor before, but generally I would use streaming APIs to fetch a few rows at a time and append them to a csv file, yielding each iteration if necessary to allow other work to complete.
That way you only have one chunk of data in memory at any time, and you can stream the download as well
lambda + s3 sounds like a nice way to offload it and not worry about memory and yielding