CollectionFS, transformWrite and callbacks

I’m building an app that stores ebooks and uses the ebook-convert NPM module to render them in a more web-friendly format. ebook-convert calls child_process.spawn to spin up the Calibre command line utility of the same name. When done, it emits an end/exit event.

I’m attempting to use CFS to store the converted ebooks. CFS has a transformWrite function that, according to the docs, accepts readStream and writeStream arguments and needs to pipe the former to the latter before the function returns. Unfortunately I don’t see how to make these streams work with child_process, which is not a stream. I have code that eventually pipes the converted ebook output to writeStream, and can confirm that on the filesystem, but it happens when the child_process emits the end event, which I assume happens after the function returns.

Can anyone recommend a solution? This is a very low-volume app (Sandstorm, likely only used by 1 person at a time) so absolute worst case scenario I can block the process. But I’d rather not. Am I missing a callback-based API in CFS for this functionality? Is there some way to convert this workflow to callback-based? Is there a better file collection that supports transforming files before they’re written? vsivsi:file-collection doesn’t document that functionality in its README so I assume it doesn’t, but I might be wrong.

Here’s my current code, using the temp and ebook-convert NPM modules. I’ve confirmed that the file is written correctly in the temporary directory but it never hits the database. I assume this is because the function returns before the pipe can happen:

  transformWrite: (fileObj, readStream, writeStream) => {
    const tempDir = temp.mkdirSync()
    const upload = fs.createWriteStream(tempDir+"/"
    const baseName = upload.path
    const htmlzName = baseName+".htmlz"
    const htmlz = convert({
      source: baseName,
      target: htmlzName
    // htmlz.on("error", (err) => console.error("Error", err.toString()))
    // htmlz.on("message", (msg) => console.log(msg.toString()))
    // htmlz.on("data", (data) => console.log(data.toString()))
    htmlz.on("exit", (code) => {
      if(code == 0) {
        const target = fs.createReadStream(htmlzName)

Did you manually install Calibre on the server like you did on your local machine?

Yes, but that’s not the issue. The conversion finishes and I can see the
file on the filesystem, but CFS’ stream-based design doesn’t seem
compatible with the ebook-convert module.

Sorry I should have read more closely, I realize now that’s not the issue. Just used to myself and others often forgetting that and from the title and glazing over thought you might have too :stuck_out_tongue:

I personally don’t have experience w/ the ebook-convert module.