UPDATED: CFS:s3 saves file to s3 but does not insert into images collection

help
1

UPDATE:

Two changes to my application code were able to resolve this problem for me.

  1. I purged my database models, eliminating all previous collections that were created from CFS:s3 under different settings and drivers.

  2. I changed the policy rules on my s3 bucket to allow public reads

Fortunately I was in a place where both of these were acceptable during development, and a public-read bucket is acceptable for prod, so it works out… for me. Best wishes to anyone who encounters this problem in the future.

/client/client_models/images.js

var imagesStore = new FS.Store.S3("handstack-test", { bucket: "handstack-test"});

Images = new FS.Collection("images", {
    stores: [imagesStore],
    filter: {
        allow: {
            contentTypes: ['image/*']
        }
    }
});

/server/lib/server_models/images.js

var imagesStore = new FS.Store.S3("handstack-test", {
    accessKeyId: Meteor.settings.private.aws.key,
    secretAccessKey: Meteor.settings.private.aws.secret,
    bucket: "handstack-test",
    region: "us-east-1" /**,
    beforeWrite: function(fileObj) {
        fileObj.size(64, {store: "imagesStore", save: false});
    },
    transformWrite: function(fileObj, readStream, writeStream) {
        gm(readStream, fileObj.name()).resize('64', '64').stream().pipe(writeStream)
    }*/
});


Images = new FS.Collection("images", {
    stores: [imagesStore],
    filter: {
        allow: {
            contentTypes: ['image/*']
        }
    }
});

Images.allow({
    insert: function () {
        return true;
    },
    update: function () {
        return true;
    },
    remove: function () {
        return true;
    },
    download: function(){
        return true;
    }
});

Making the following call on the client

Images.insert(image, callback);

Results in the following websocket traffic being sent:

["{\"msg\":\"method\",\"method\":\"/cfs.AWSImages.filerecord/update\",\"params\":[{\"_id\":\"YuP9PbuGAtD2mheiK\"},{\"$set\":{\"chunkSize\":2097152,\"chunkCount\":0,\"chunkSum\":1}},{}],\"id\":\"3\"}"]	200	
10:27:36.529
["{\"msg\":\"method\",\"method\":\"/cfs.AWSImages.filerecord/update\",\"params\":[{\"_id\":\"YuP9PbuGAtD2mheiK\"},{\"$push\":{\"userIds\":\"BuHRXBWmwQN6PTrp9\"}},{}],\"id\":\"4\"}"]	181	
10:27:36.555
a["{\"msg\":\"result\",\"id\":\"3\",\"result\":1}"]	51	
10:27:36.599
a["{\"msg\":\"updated\",\"methods\":[\"3\"]}"]	46	
10:27:36.606
a["{\"msg\":\"result\",\"id\":\"4\",\"result\":1}"]	51	
10:27:36.648
a["{\"msg\":\"updated\",\"methods\":[\"4\"]}"]	46	
10:27:36.737
h	1	
10:27:41.513
a["{\"msg\":\"ping\"}"]	23	
10:27:56.567
["{\"msg\":\"pong\"}"]

And will save the image to s3, but not to my Images collection.

It will save properly to CFS’s ‘cfs.images.filerecord’ collection.

And here are the packages I’m using:

# Meteor packages used by this project, one per line.
# Check this file (and the other files in this directory) into your repository.
#
# 'meteor add' and 'meteor remove' will edit this file for you,
# but you can also edit it by hand.

meteor-base             # Packages every Meteor app needs to have
mobile-experience       # Packages for a great mobile UX
mongo                   # The database Meteor supports right now
session                 # Client-side reactive dictionary for your app
jquery                  # Helpful client-side library
tracker                 # Meteor's client-side reactive programming library

standard-minifiers      # JS/CSS minifiers run for production mode
es5-shim                # ECMAScript 5 compatibility for older browsers.

angular@1.3.1
angularui:angular-ui-router
accounts-password
angularutils:pagination
tmeasday:publish-counts
check
email
fourseven:scss
driftyco:ionic
civilframe:angular-jade
reywood:publish-composite
momentjs:moment
jasonaibrahim:angular-moment
digilord:faker
aldeed:collection2
http
cfs:standard-packages
cfs:gridfs
danialfarid:ng-file-upload
alexk111:ng-img-crop
vitalets:angular-xeditable
mikowals:batch-insert
deanius:promise
sebastianilves:angular-chart-js
meteorhacks:npm
meteorhacks:picker


npm-container
angularui:angular-google-maps
cfs:s3
cfs:graphicsmagick
2

This may be the same CORS issue as reported here. To solve this, CollectionFS will have to set the Access-Control-Allow-Origin header.

3

@martijnwalraven

Dug through those issues and I’m not quite sure what you mean.

I did however add the following code just incase:

Meteor.startup(function () {
    WebApp.rawConnectHandlers.use(function(req, res, next) {
        res.setHeader("Access-Control-Allow-Origin", "*");
        res.setHeader('Access-Control-Allow-Methods', '*');
        res.setHeader('Access-Control-Allow-Headers', 'Content-Type');
        const originalSetHeader = res.setHeader;
        res.setHeader = function (key, value) {
            if (key === 'Access-Control-Allow-Origin'
                || key === 'Access-Control-Allow-Methods'
                || key === 'Access-Control-Allow-Headers') {
                console.warn(`Ignoring setting of ${ key } to ${ value }`);
                return;
            }
            return originalSetHeader.apply(this, arguments);
        };
        next();
    });
});

And it didn’t seem to fix the issue.

4

I don’t think I understand everything you’re trying to do here, but I think the issue is that CollectionFS actually sets Access-Control-Allow-Origin itself and probably overrides your setting. This is the line that needs to be changed, either by setting a wildcard * or by echoing the origin in the request.

5

@martijnwalraven

My understanding is that CFS:S3 is supposed to, upon Images.insert(image), save the file to s3 and insert a document into the Images collection with a reference to the url stored on s3. However when I make the call to Images.insert it will save properly to s3 but not insert that document into the collection.

With the Meteor.startup block I set the headers how you describe, and replace the .setHeader() function so that those headers will never be overwritten. The console.log statement actually gets called everytime I try to upload the file, suggesting that cfs:s3 is actually trying to overwrite the header on every call. Weird.

Thanks for all the help btw, this problem kept me up at night and it’s still killing me.

6

I think the issue might be that Access-Control-Allow-Methods does not allow wildcards. You have to specify the actual methods allowed.

Setting Access-Control-Allow-Origin should be enough though, the other headers should be set by CollectionFS.

7

I amended my start-up script

Meteor.startup(function () {
    WebApp.rawConnectHandlers.use(function(req, res, next) {
        res.setHeader("Access-Control-Allow-Origin", "*");

        const originalSetHeader = res.setHeader;
        res.setHeader = function (key, value) {
            if (key === 'Access-Control-Allow-Origin') {
                console.warn(`Ignoring setting of ${ key } to ${ value }`);
                return;
            }
            return originalSetHeader.apply(this, arguments);
        };
        next();
    });
});

W20160227-13:04:46.659(-8)? (STDERR) Ignoring setting of Access-Control-Allow-Origin to http://meteor.local
W20160227-13:04:54.895(-8)? (STDERR) Ignoring setting of Access-Control-Allow-Origin to http://meteor.local


Are the console logs from that block.

Didn't change anything.
8

@martijnwalraven

I’ve updated my top post with what was a resolution in my particular circumstance. It appeared to not be a problem with CORS, though I’ve yet to remove the CORS piece so it might’ve helped.

Anyone who sees this in the future, it might be better to write your own s3 driver or hack one together using these two meteor packages:

meteorhacks:npm
meteorhacks:picker

Which allow you to create an expressJS style API endpoint where I presume you can do a traditional file upload and send it off to s3 like a traditional web server.