Best way to store rich media server-side?

In my Cordova app, the user may take pictures or videos with the device camera and upload them. So I need a reliable way of dealing with rich media on the server side.

My first - prototypical - approach was to store the data as base64 in Mongo collections (for each media type), but I don’t think this is a good solution and would prefer a kind of server-side file storage that can also be accessed via http:// URLs.

I came across CollectionFS, which seems very promising. However, they clearly state in the GitHub repository: ‘… don’t use in production yet.’ This is not really what I am looking for (maybe it’s just a safety statement, but who knows?).

So I am wondering if there are any other approaches to this scenario out there that allow upload and retrieval of rich media files?

It would also be perfect if this solution would be able to cache these files locally in a Cordova app, just like MiniMongo does, to provide a better end-user experience and buffer connection losses.

One last question I have in mind regarding this: Is there any restriction on the size of Session variables? Background: As I said earlier, I’m currently storing images (only) inside MongoDB as base64 data. If a user enters a template using these images, I retrieve them from the database and store them - together with the overall model document - in a session variable. But I doubt this is a good idea, since I assume these Session variables are being stored in browser-side stores like cookies or HTML5 web storage. This is also the reason why I am looking for a solution that would allow me to retrieve the images via an HTTP URL.

https://atmospherejs.com/edgee/slingshot

@sikanx: Thanks for the fast reply. Slingshot looks promising.

Slingshot is great. We actually use Uploadcare as a service. There’s a decent package for integration (think it’s by @natestrauser) .

The provide an upload widget which is cool, but better still they give you a CDN which would allow you to call down smaller versions saving on speed.

Might be more than you’re looking for, but check out the free plan and have a play around!

@sgoudie: Thanks for the link to the uploadcare service, sounds interesting to me. A question on this: does uploadcare also support private CDN requests, or is this only useful for public CDNs? Background: To some extent, my app uses mechanisms like facebook’s, i.e. some content shall only be visible to “friends”, while others should be publicly available. Hence, I need a way to restrict access to files stored in the CDN to certain users (the owner and the owner’s friends). It would be awesome to have a CDN in place here, to allow for a global rollout without performance issues.

I now made a kind of deep-dive into both the CollectionFS and Slingshot documentations. Would like to share my thoughts here, in case someone else is facing the same scenario. Here are my impressions so far:

CollectionFS:

Pros: Has a very broad and thorough approach, supports a whole bunch of backends through plugins. Also offers a lot of meaningful helpers, e.g. for transforming data on the fly, I like the overall approach very much. Especially because it can support a smooth transformation from a file-based storage to a cloud-based variant. This would also allow to store files locally in a Cordova application, which is nice for my scenario.

Cons: I could not find any information about CDN support so far. There’s one issue in the Github tracker on this, but this hasn’t been answered by now. They also state clearly that CollectionFS should not be used in productive scenarios yet. I’m still wondering if should be taken too seriously, since the adoption seems to be quite good so far.

Slingshot:

Pros: Interesting approach, as it allows clients to send files directly to a cloud-based service, without the need to stream them to the app server first. For apps using uploads quite frequently, this will reduce server load significantly.

Cons: As it is tightly coupled to a cloud service, there is no convenient way to set-up a local file-based cache, e.g. for all media the user is the owner of. So in a mobile app, one would have to mix this technology with other technologies like CollectionFS or Cordova’s file plugins. Latency compensation for uploads is supported, though. I also could not find any information how to support public or private CDNs with Slingshot, although the Rackspace integration seems to be somewhat related to CDNs.

As an alternative, uploadcare.com seems to be an interesting service, and I also found filepicker.com which is providing a similar service and has a quite decent customer base (but is way more expensive than uploadcare). EDIT: Besides the more attractive pricing, uploadcare includes a CDN, whereas filepicker requires you to set-up a CDN yourself on Amazon Cloudfront. The uploadcare CDN does not support private CDNs, however. So overall, I’d say that going for one of these services is not a good choice for me, as I would have to set-up my CDN anyway - and there’s not so much benefit left compared to implementing a storage based on CollectionFS or Slingshot (besides the nice upload widgets, of course, and some of the out-of-the-box image processing options).

Yet, I have to admit, I’m still undecided :smile:

I highly recommend uploadcare or filepicker. Personally, I really like the nice upload widgets that both provide as well as the on demand image conversion. Filepicker used to be my go to option, but they raised their prices for the plans that include image conversion, which put them out of my client’s price range.

For the private CDN, is the obscurity of the filename not enough to protect the assets? I don’t think you can list the directory and the file ids are rather unguessable. If you control publication of this information (file ids) from the app, would that do the trick?

re-read your first post and thought of something else…

you may want to consider doing something like this to directly upload from a cordova device to uploadcare

Template.photoButtons.events({
    'click #take-photo': function(event, template) {
        var self = this;
        uploadcareCaptureImageUpload(function(error, fileInfo){
            if(error){
               //alert error
            }else{
                //do something with fileInfo object - usually save the id onto an object
            }
        });
    }
});


//helper function to be reused
uploadcareCaptureImageUpload = function(cb) {
    var callback = cb;

   

     var cameraSuccess = function(imageURI) {
            var options = {
                mimeType: 'image/jpeg',
                fileName: "image-upload-" + Meteor.userId() + "-" + Date.now() + ".jpg",
                params : {
                    "UPLOADCARE_PUB_KEY": <UPLOADCARE-PUBLIC-KEY>,
                    "UPLOADCARE_STORE": 1    
                }
            };
    
            var ft = new FileTransfer();
    
            ft.onprogress = function(progressEvent) {
                if (progressEvent.lengthComputable) {
                    console.log(progressEvent.loaded / progressEvent.total);
                }
            };
            
            ft.upload(imageURI, "https://upload.uploadcare.com/base/", function(result) {
                var ucFile = uploadcare.fileFrom('uploaded', JSON.parse(result.response).file);
                ucFile.done(function(fileInfo) {
                    callback(null, fileInfo);
                });
    
            }, cameraFail, options);
        };
    
    
        var cameraFail = function(message) {
            callback(message);
        };
    
        navigator.camera.getPicture(cameraSuccess, cameraFail, {
            quality: 90,
            destinationType: Camera.DestinationType.FILE_URI,
            sourceType: Camera.PictureSourceType.CAMERA,
            allowEdit: false,
            encodingType: Camera.EncodingType.JPEG,
            saveToPhotoAlbum: false
        });

};

I have 2 apps that I’m currently working on using this and it works really well. With slightly different code, you can also capture videos.

You can also get a preview from the local filesystem, but I would not recommend using base64 data as it can really bog down lower performing devices. To access the filesystem you can use a cdvfile:// url or a httpd server as a local proxy (see https://github.com/meteor/meteor/issues/3799)

Thanks for this info!

When I have tried to use FILE-URI instead of DATA-URL for navigator.camera.getPicture I got “not allowed to load local resource”. How did you access FILE-URI?

Actually my intent was to use video with navigator.device.capture.captureVideo, however I got MediaFile with fullPath and localURL and both just give me 404 if I try to access it with meteor. I have tried your natestrauser:cordova-file-server, still can not make it working

For example, I have cdvfile://localhost/sdcard/DCIM/Camera/file.mp4 from cordova, but it is not accessible within meteor. It is really frustrating

AFAIK, you cannot access the camera picture as a file URI, due to security constraints in the web views. But you can upload it to a server, if you convert it from a data URL to a so-called “Blob”. slingshot accepted this without any problems, but I had problems with CollectionFS. This was one of the reasons why I switched over to slingshot.

For the conversion, I am using the method below. I tried many different shims from several Stackoverflow posts, and this version worked best :smile:

var dataURLToBlob = function(dataURL) {
  var BASE64_MARKER = ';base64,';
  var parts, contentType, raw;
  if (dataURL.indexOf(BASE64_MARKER) === -1) {
    parts = dataURL.split(',');
    contentType = parts[0].split(':')[1];
    raw = decodeURIComponent(parts[1]);
    return new Blob([raw], {
      type: contentType
    });
  }
  parts = dataURL.split(BASE64_MARKER);
  contentType = parts[0].split(':')[1];
  raw = window.atob(parts[1]);
  var rawLength = raw.length;
  var uInt8Array = new Uint8Array(rawLength);
  for (var i = 0; i < rawLength; ++i) {
    uInt8Array[i] = raw.charCodeAt(i);
  }
  return new Blob([uInt8Array], {
    type: contentType
  });
}; 

I can do pictures with just dataURL (put it in Session, set on a canvas, read from the canvas and voila - I have an image), and upload to cloudinary. But still can not find a way how I can do videos. For videos I don’t have data, I only have a path (file) or a local url (cdvfile).

I am not sure it is a webview limitation, cordova by itself handles it with no issue, it is that I can not use meteor to access it after that