Your approaches to store large amounts of user-generated content?


#1

I’m building an app that will the users to store a considerable amount of self-generated content. It’s images in the first place, but it may be extended to videos later on.

So I am wondering which backend services the community is using for this and what your experiences are. I’ve already implemented an S3 connection in combination with the slingshot package, but the response times I get from AWS are sometimes very long.

I’ve also considered using CollectionFS/GridFS, as this seems to be a scalable architecture due to the sharding possibilities, but I don’t know how good this actually performs in larger scales.

So, how are you doing it, and what are your experiences with your approach (performance, costs, other things to consider)? Any input is welcome. Thanks.


#2

I’m using Slingshot, and it’s blazing fast. Curious that you’re experiencing issue with responsiveness. I gave CollectionFS a try and didn’t like it, I felt it was overly complex compared to Slingshot.


#3

Thanks for being the first replying to my question :thumbsup:

Regarding Slingshot: Yes, the upload itself is fast. But I’m having trouble with the download from S3. Sometimes, it is very fast. But on some days, it takes incredibly slow. You can even watch the images to build up. And this has nothing to do with responsiveness, as this also happens with images that have been stored a while ago.

I agree with you with CollectionFS. I have implemented CollectionFS/S3 during the last days, because I also needed a server-only upload. And I had as many problems doing this as with CollectionFS/GridFS. Main reason is missing and outdated documentation. For instance, there is no description in the main docs how to get an URL to an uploaded file after the upload. Instead, it has a long description about how you can use image transforms. Which is nice in itself, but accessing the uploaded image is a core feature compared to that, if you ask me.


#4

This was the same issue I had with cfs too but it seems to do the job well. You could always use a combination of cfs and slingshot though, nothing stopping you from doing that! Maybe use cfs for local storage and slingshot for s3?

I don’t know why you’re having responsiveness issues though… Do sorry I cannot help there! :frowning:


#5

Strange. So this sounds like an S3 issue. My S3 downloads are blazing fast. And you’re not using the limited availability option?


#6

I don’t know, I’m still on the evaluation period.


#7

What do you mean by that?