How can I use GridFS to load a file on the server after I create it server side in a Meteorjs app

Hi all,

How can I use GridFS to load a file on the server after I create it server side. I do have code to load files client side using GridFS but I am having a hard time changing the code to work server side in the case where the file is created on the fly on the server. It makes sense to load it server side because hundreds of users will be creating these files on the fly as they access my meteor app.

I want to be able to load any file and zip files. So the code must work for zip files as well. Please help.

Thanks
B.

I’m not sure what you mean when you said “load a file”.
If you want to store a file in GridFS (data stored in mongodb) you can do do something like this:

      import Grid from 'gridfs-stream';
      import fs from 'fs';
      // ...
      const gfs = Grid(
        MongoInternals.defaultRemoteCollectionDriver().mongo.db,
        MongoInternals.NpmModule,
      );
      const writeStream = gfs.createWriteStream({ filename, metadata: someMetaData });
      fs.createReadStream(fileSourcePath).pipe(writeStream);
      writeStream.on('close', (gfsFile) => console.log(gfsFile));
      writeStream.on('error', (err) => console.log(err));
``

My client code inserts files into images collection similar to this code. Scroll down to see final code altogether. I understand the code makes use of client side hooks (not sure if the same hooks can work server side). How do I make the same code work server side? The code inserts the file into images, fs.chunks and fs.files collections. It works brilliantly client side. But they did not share how to make the same code work server side.

My code looks as follows:

import { FilesCollection } from 'meteor/ostrio:files';
import Grid from 'gridfs-stream';
import fs from 'fs';

let gfs;
if (Meteor.isServer) {
  const mongo = MongoInternals.NpmModules.mongodb.module; // eslint-disable-line no-undef
  gfs = Grid(Meteor.users.rawDatabase(), mongo);
}

Images = new FilesCollection({
  collectionName: 'images',
  //allowClientCode: false,
  debug: false,
  onAfterUpload(image) {
    // Move file to GridFS
    Object.keys(image.versions).forEach(versionName => {
      const metadata = { versionName, imageId: image._id, storedAt: new Date() }; // Optional
      const writeStream = gfs.createWriteStream({ filename: image.name, metadata });

      fs.createReadStream(image.versions[versionName].path).pipe(writeStream);

      writeStream.on('close', Meteor.bindEnvironment(file => {
        const property = `versions.${versionName}.meta.gridFsFileId`;

        // If we store the ObjectID itself, Meteor (EJSON?) seems to convert it to a
        // LocalCollection.ObjectID, which GFS doesn't understand.
        this.collection.update(image._id, { $set: { [property]: file._id.toString() } });
        this.unlink(this.collection.findOne(image._id), versionName); // Unlink files from FS
      }));
    });
  },
  interceptDownload(http, image, versionName) {
    // Serve file from GridFS
    const _id = (image.versions[versionName].meta || {}).gridFsFileId;
    if (_id) {
      const readStream = gfs.createReadStream({ _id });
      readStream.on('error', err => { throw err; });
      readStream.pipe(http.response);
    }
    return Boolean(_id); // Serve file from either GridFS or FS if it wasn't uploaded yet
  },
  onAfterRemove(images) {
    // Remove corresponding file from GridFS
    images.forEach(image => {
      Object.keys(image.versions).forEach(versionName => {
        const _id = (image.versions[versionName].meta || {}).gridFsFileId;
        if (_id) gfs.remove({ _id }, err => { if (err) throw err; });
      });
    });
  },
});



if (Meteor.isServer) {
	Images.denyClient();
}

Example of inserting a file client side is

		'change .persInfoFileInput': function (e, template) {
			var username = this.username;
			var user = Meteor.users.findOne({username: username});
			var id;
			var fileName;
			id = user._id;
		//event.preventDefault();
			if (e.currentTarget.files && e.currentTarget.files[0]) {
			  // We upload only one file, in case
			  // there was multiple files selected
			  var file = e.currentTarget.files[0];
			  var fileName = "User_" + id + "_" + "Details" + "_"

			  Images.insert({
				file: file,
				fileName: fileName + e.currentTarget.files[0].name,
				onStart: function () {
				  template.currentUpload.set(this);
				},
				onUploaded: function (error, fileObj) {
				  if (error) {
					alert('Error during upload: ' + error);
				  } else {
				  }
				  template.currentUpload.set(false);
				},
				streams: 'dynamic',
				chunkSize: 'dynamic'
			  });
			}
		},

Example of deleting file client side is:

		"click .deletePersInfoFile": function (event) {
			var id = event.target.name;
			Images.findOne(id).remove();
		}

Displaying the file looks something like this

		persInfoImages: function () {
			var username = this.username;
			var user = Meteor.users.findOne({username: username});
			var id;
			var partOfFileName;
			id = user._id;
			partOfFileName = ".*User_" + id + "_" + "Details_.*";
			//search = new RegExp(partOfFileName, 'i');
			//var imgs = Images.find({"name":search});
			var imgs = Images.find({"name":{$regex:partOfFileName}});
			//var imgs = Images.find({"copies.images.name":{$regex:partOfFileName}})
			//var imgs = Images.find();
			return imgs // Where Images is an FS.Collection instance
		}

Ofcourse the above are the invoked methods for inserting, removing and dispaying. I omitted the html side of things.

@dr.dimitru Can you help?

@beonne the code included server side code (FilesCollection is isomorphic). You can check for uploaded binaries using meteor mongo and in the Mongo shell running

db.fs.files.find()

and

db.fs.chunks.find()

Edit: check the onAfterUpload function. This is on the server where the uploaded file is moved from the filesystem to GridFs

@jkuester , I did suspect the code as defined by Images = new FilesCollection({ … as per above can be used on the server. My attempt to do Image.findOne() works but when I do Images.insert() I get the error: “Images.insert is not a function”. I was hoping by doing Images.insert() that onAfterUpload will be invoked, but I get an error instead. Here is my attempt with comments at writing the server side code:

			var fs = require('fs');
			const HummusRecipe = Npm.require('hummus-recipe');
			const AdmZip = Npm.require('adm-zip');
                        //Here I am just creating a pdf file
			const pdfDoc = new HummusRecipe('new', 'output.pdf',{
				version: 1.6,
				author: 'Digitrade',
				title: 'Your purchased vouchers',
				subject: 'List of purchased vouchers'
			});
			

			pdfDoc.createPage('letter-size')
			pdfDoc.text('start from here', 0, 0)
			pdfDoc.text('next line', 0, 20)
			pdfDoc.endPage()
			pdfDoc.encrypt({
					userPassword: '123',
					ownerPassword: '123',
					userProtectionFlag: 4
				})
			pdfDoc.endPDF();
			
			

			const file = new AdmZip();
                        //Here I am just zipping the pdf file  
			file.addLocalFile('./output.pdf');
			file.writeZip('output.zip');
			console.log(process.cwd());
			
                        //From here is my attempt at inserting the file using GridFS
			var id;
			var fileName;
			id = Meteor.userId();
			var file1;
			fileName = "User_" + id + "_" + "Details" + "_" + 'output.pdf'
			
			var data1 = fs.readFileSync(process.cwd() + '/output.pdf')
			//1. Not sure how to pass the file as it is passed on the client, so I hard coded the JSON that I see is passed on the client
			var data2 = {lastModified:1631301228000,lastModifiedDate:"Fri Sep 10 2021 21:13:48 GMT+0200 (SAST) {}",name:"output.pdf",size:10905,type:"application/pdf",webkitRelativePath:""};
			console.log(Images);
			//2. Images.findOne works, but Images.insert or Images.insertOne as per below do not work
			//3. I get the error: Images.insert is not a function
			var im = Images.findOne({})
			console.log(im);

			//4. Not sure how to mimic the Images.insert on the client. Not sure how to mimic the working code on the client
			Images.insert({
				file: data2, //5. Do not know how to pass the same data as per the JSON on the client
				fileName: fileName,
				onStart: function () {
				  data1 //6. Not sure how to mimic the client code
				},
				onUploaded: function (error, fileObj) {
				  if (error) {
					alert('Error during upload: ' + error);
				  } else {
				  }
				  return true //7. Not sure how to mimic the client code
				},
				streams: 'dynamic',
				chunkSize: 'dynamic'
			});
			

I am sure my attempt is wrong somewhere (also look at my comments in the code to see some of the areas I am experiencing difficulty). From my attempt are you able to spot where I am going wrong and correct please? So my invoking of Images.insert() server side does not work as I am getting the error above. If you want me to provide the html side of things at-least for the client code I supplied earlier so that you build a little app just to test the client before you correct my server let me know.

@beonne

  1. I recommend to use native mongodb driver for GridFS integration, take a look at the latest docs;
  2. After a new file is created on the server use .addFile(), pay attention to setting proceedAfterUpload argument to true (after callback function);
  3. After file is uploaded from the client extend user’s object with object containing only required fields for fileURL helper or .link() method returned from onUploaded() hook; Alternatively you can pass user’s _id into meta.owner and later set index on this field for quick and efficient Mongo query.

@aureooms thank you for mentioning

2 Likes

@dr.dimitru, and everyone thanks for your help. I have no further questions for now. The documentation is clear at first glance. I’ll shout if I get stuck. Thanks.

2 Likes

@dr.dimitru and everyone, thanks it worked. I posted another question as I can’t get Reactjs to download the files as I used to do it in Iron-router. You can help in this regard if it is also up your alley. Thanks again.