How to stream a large file from meteor server to client?

help
1

We’re storing file uploads on S3 with edgee:slingshot, meaning the files are uploaded directly from client -> S3.

However for download, we pass through our server, meaning S3 -> server -> client.

Right now, when the user clicks on download, we just use a simple meteor method:

Meteor.methods({
  downloadFile: (key) => s3.getObject(key), // Simplified for demonstration purposes
})

This properly returns the file on the client, which is then downloaded with the file-saver library to his computer.

The problem with this is, when users are downloading large files, this completely crashes the app with this error:

2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR Killed
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! code ELIFECYCLE
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! errno 137
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! my-meteor-app@ start: `node launcher.js`
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! Exit status 137
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR!
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! Failed at the my-meteor-app@ start script.
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! A complete log of this run can be found in:
2018-10-16 15:21:05 [APP/PROC/WEB/0] ERR npm ERR! /home/vcap/app/.npm/_logs/2018-10-16T13_21_05_399Z-debug.log
2018-10-16 15:21:05 [APP/PROC/WEB/0] OUT Exit status 137 (out of memory)

I presume it is caused by what is described in this stack overflow issue: https://stackoverflow.com/questions/20875314/download-large-file-from-node-http-server-the-server-out-of-memory

So has someone already written meteor methods that return a node stream? or something similar to stream the file to the client ?

2

this also returns an object with the method createReadStream so you can essentially do s3.getObject(key).createReadStream().pipe(response) where response is a typical http response object provided to you by meteor’s raw connect handlers, meaning, a rest endpoint as opposed to a method.

And since this is a piped stream, your server will not have to process the file, instead pass it directly to your http response.

A meteor would have returned the data over meteor’s websocket but downloading/uploading large chunks of data over websockets is not ideal, hence the http endpoint.

You can still incorporate a meteor method into this to handle some business logic, check for permissions etc and then return a unique http url - such as /files/:fileId - for the actual file download to kick in.

Edit: here’s the reference to the stream for convenience, notice that getObject returns an AWS.Request.

1 Like
3

you could also create a short-time download url:

export const getS3Params = Key => ({
  Bucket: process.env.AWS_BUCKET,
  Key: Key
});

export const createS3 = () =>
  new AWS.S3({
    region: 'eu-central-1',
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    signatureVersion: 'v4',
  });
export const getDownloadUrl = Key =>
  createS3().getSignedUrl('getObject', {
    ...getS3Params(Key),
    Expires: 180,
  });
1 Like
4

This is also a very straight forward approach, perhaps the easiest, but note that the expiry is only ttl based and the url is public so it might still be preferable to at least put it behind a connect proxy middleware that does basic checks such as a valid user session and maybe even fully expire the url after successful download.

1 Like