Ensuring fetch() (was find()) times out after a period

[pmcochrane]

I’m trying to get my Collection finds (on the server) to timeout after a period to ensure that I do not accidentally pass in bad search criteria on a large collection.

e.g. ColPatients.find({}, { maxTimeMs: 30000}).fetch()

From the guide, the maxTimeMs option should kill the find after 30 seconds.

However, in my case, the find is not timing out and simply uses 100% of the server until it runs out of javascript heap space. Has anyone got the maxTimeMs option to work in practice?

I know a fix is to limit the query or ensure that bad search criteria is not passed to the find but I’d like the reassurance of it automatically timing out rather than waiting on the server to crash if something unusual happens.

[peterfkruger]

The option maxTimeMs only makes sure that the query times out, not the fetch(). Are you sure what’s taking so long is actually the query (find), or may it be that the query returned quickly, and the subsequent fetch or some related operation on the fetched data takes enormously long?

[pmcochrane]

Doh. Well spotted. You are 100% correct. It’s the fetch that is taking the time. Didn’t notice that.

Any idea how to match the fetch timeout if it runs on too long? I can’t see any options in the fetch documentation so far

Thanks

[peterfkruger]

You could use Cursor#forEach instead of fetch().

Then, within the callback you can measure the time, and if a threshold is exceeded, you can throw a Meteor.Error, which you can catch either within the same Meteor method (if that makes sense), or even on the client.

[pmcochrane]

Thanks. I’d already started similar from looking at the source for fetch in mini mongo. I’ve padded it out to a function. Some bits are specific to my codebase (inf for console logging) but may be useful to others.

// Fetch data from a mongo cursor with a timeout
function fetchWithTimeout(curs: Mongo.Cursor<any>, collectionName: string, searchCriteria: any, searchProjection: any, inf: LogInfoData, timeout=30000 ): any[] {
    let intervalHandle: number;
    let retval: any;
    let cancelFetch=false;
    const stime=moment();
    try {
        intervalHandle = Meteor.setInterval( () =>  {
            const dur = moment().diff(stime, 'ms');
            consoledebug(inf.pre, "LONG fetch operation still running after "+dur+"ms:", collectionName, JSON.stringify(searchCriteria));
            if (dur>timeout) {
                if (intervalHandle) { Meteor.clearInterval(intervalHandle); }
                cancelFetch=true;
                consoleerror(inf.pre, "Fetch timed out: collection:", collectionName, "searchCriteria:", JSON.stringify(searchCriteria), "searchProjection:", searchProjection );
                throwMeteorError(inf, "Failed to execute find (timed out) against the collection: "+collectionName);
            }
        }, 5000);
        // Fetch the data from the mongo cursor
        retval=[];
        curs.forEach( (doc: any) => {
            retval.push(doc);
            if (cancelFetch) {
                if (intervalHandle) { Meteor.clearInterval(intervalHandle); }
                consoleerror(inf.pre, "Fetch loop timed out: collection:", collectionName, "searchCriteria:", JSON.stringify(searchCriteria), "searchProjection:", searchProjection );
                throwMeteorError(inf, "Failed to execute fetch (timed out) against the collection: "+collectionName);
            }
        });
        if (intervalHandle) { Meteor.clearInterval(intervalHandle); }
        return retval;
    } catch (error) {
        if (intervalHandle) { Meteor.clearInterval(intervalHandle); }
        consoleerror(inf.pre, "Fetch loop exception: collection:", collectionName, "searchCriteria:", JSON.stringify(searchCriteria), "searchProjection:", searchProjection );
        throwMeteorError(inf, "Fetch loop exception against the collection: "+collectionName);
    }
    if (intervalHandle) { Meteor.clearInterval(intervalHandle); }
    consoleerror(inf.pre, "Fetch loop unknown error: collection:", collectionName, "searchCriteria:", JSON.stringify(searchCriteria), "searchProjection:", searchProjection );
    throwMeteorError(inf, "Fetch loop unknown error against the collection: "+collectionName);
}

The usage of this function is:

const data = fetchWithTimeout((<any> ColPatients).find(searchCriteria, searchProjection),
                            collectionName, searchCriteria, searchProjection, inf);

Basically do a find to get the cursor and pass it to the function. The other parameters are for logging context specific errors to assist in debugging

[paulishca]

As per Meteor documentation here: Collections | Meteor API Docs

"
maxTimeMs Number

(Server only) If set, instructs MongoDB to set a time limit for this cursor’s operations. If the operation reaches the specified time limit (in milliseconds) without the having been completed, an exception will be thrown. Useful to prevent an (accidental or malicious) unoptimized query from causing a full collection scan that would disrupt other database users, at the expense of needing to handle the resulting error.
"

In theory, this should be enough for you. I set this on one of my find and , yes, got the exception: