Fast in-memory search with Meteor

[ottobonn]

Hi all!

At Qualia we’ve been working on improving search in our Meteor app. We were using EasySearch to construct MongoDB queries based on regex selectors, and we were having trouble getting search to behave the way we wanted. In particular, it was hard to prioritize exact matches for the search queries, and there wasn’t an easy way to use one search query to look across multiple fields on the same document in the collection.

We decided to maintain a projection of our collection in memory on the Meteor server, and then use plain JavaScript to search over it. The result is pretty fast and really flexible!

Check out our blog post about it, and the new libraries, MappedCollection for projecting a collection into memory and lasr for search!

[coagmano]

This looks great! Thanks for sharing this

[vigorwebsolutions]

Cool strategy! I’m curious about the scalability? The in-memory cache seems like it would be pretty scalable but I’m also wondering about the expense of that cursor.observe and how closely that is tied to the size of the collection.

Currently I’m implementing something very similar to this: https://web.archive.org/web/20170609122132/http://jam.sg:80/blog/efficient-partial-keyword-searches/ as a solution for bigger scaling and cross-collection searches.

I’m also wondering how tightly coupled you think your strategy is with your specific use case?

Good to see you guys always pushing the boundaries! =)

[claytonzaugg]

Not sure how often you’re on here or check notifications, but I had some questions about MappeCollection and lasr for a project I’m working on for our company. I wanted to ask more detailed questions about getting it functioning properly, as I can’t seem to figure it out or am missing something so blindingly obvious it’s scary.

Please let me know if you’re about and still keeping up with this project so I can ask away!

Thank you

[ottobonn]

We’re still using it in production; let me know how I can help!

[claytonzaugg]

For example, here is the code I have in mapped-collection.js in my server folder

import { MappedCollection } from 'meteor/qualia:mapped-collection';
import { lasr } from 'meteor/qualia:lasr';
import '../imports/api/items/collection-items.js';

const mappedFields = [
	'MFR_FULLNAME',
	'MFR_CAT_NUM',
];

const mappedItems = new MappedCollection({
	collection: Items,
	fields: mappedFields,
});

// mappedItems.ready resolves when the map is fully initialized
Promise.await(mappedItems.ready);

// mappedItems.map is a normal ES6 Map
const items = Array.from(mappedItems.map.values());

// Here, we search for some items, but you can do anything with the map!
const results = lasr.search({
	items: items,
	query: 'RAYOVAC',
	keys: ['MFR_FULLNAME'],
	limit: 10,
});

console.log(results);

But nothing console logs on the server terminal, and from what I understood, I’d see 10 items/documents with a manufacturer name of RAYOVAC in it.

[claytonzaugg]

So I tried taking the code and moving it out of the straight up server folder and included it in the api import process and I’m getting results now, but because my DB is very large, I’m getting a JS heap out of memory error. Do I need to create an index or something first with my data to help this process? Not sure how to do that, so I could use some guidance if that’s that solution.

imports/
---- api/
------ items/
--------server/
----------mapped-items.js

import { MappedCollection } from 'meteor/qualia:mapped-collection';
import { lasr } from 'meteor/qualia:lasr';
import { Items } from '../collection-items.js';

const mappedFields = [
	'MFR_FULLNAME',
	'MFR_CAT_NUM',
];

console.log(mappedFields);

const mappedItems = new MappedCollection({
	collection: Items,
	fields: mappedFields,
});

// console.log('mappedItems before promise');
// console.log(mappedItems);

// mappedItems.ready resolves when the map is fully initialized
Promise.await(mappedItems.ready);

// console.log('mappedItems after promise');
// console.log(mappedItems);

// mappedItems.map is a normal ES6 Map
const theItems = Array.from(mappedItems.map.values());

// console.log('theItems array');
// console.log(theItems);

// Here, we search for some items, but you can do anything with the map!
const results = lasr.search({
	items: theItems,
	query: 'RAYOVAC',
	keys: ['MFR_FULLNAME'],
	limit: 10,
});

console.log('### RESULST OF MAPPED COLLECTION ###');
console.log(results);

[ottobonn]

You’re using it correctly. How many items are you mapping? You might try increasing the heap memory limit for Node using this environment variable:

NODE_OPTIONS="--max-old-space-size=8192"