Meteor Mongodb deep searching

help
1

Hey everyone!
I am trying to achieve the best possible way to search between different collections.
Here is a simple scenario:

Just like WhatsApp, I want to search the chats.

Here is a very simple criteria I just created:

I have 2 collections:

  • Users
  • Chattings

Users:

[
	{
		"_id": "Xuibgsadbgsi35Gsdf",
		"fullName": "User A"
	},
	{
		"_id": "Afstg34tg4536gh",
		"fullName": "User B"
	},
	{
		"_id": "KHJDFhfs7dfgsvdfwsef",
		"fullName": "User C"
	}
]

Chatting:

[
	{
		"_id": "Phsdfyg92345sgb7651",
		"title": "Group 1",
		"chatType": "groupChatting"
	},
	{
		"_id": "YONgsa793423bD",
		"title": "All users group",
		"chatType": "groupChatting"
	},
	{
		"_id": "Uysdf345GBHsdf",
		"chatType": "oneToOneChatting",
		"ownerUserId": "Afstg34tg4536gh",
		"chatWithUserId" "Xuibgsadbgsi35Gsdf"
	},
	{
		"_id": "Fbgu283gbnap023656",
		"chatType": "oneToOneChatting",
		"ownerUserId": "KHJDFhfs7dfgsvdfwsef",
		"chatWithUserId" "Afstg34tg4536gh"
	},
	{
		"_id": "Ong7gAUDSFg7235ng",
		"title": "Group 2",
		"chatType": "groupChatting"
	}
]

Here is my query:
Chatting.find({"title": {$regex : ".*user.*", $options: 'i' }}).fetch();

Now the expected result here is:

[
	{
		"_id": "YONgsa793423bD",
		"title": "All users group",
		"chatType": "groupChatting"
	},
	{
		"_id": "Uysdf345GBHsdf",
		"chatType": "oneToOneChatting",
		"ownerUserId": "Afstg34tg4536gh",
		"chatWithUserId" "Xuibgsadbgsi35Gsdf"
	},
	{
		"_id": "Fbgu283gbnap023656",
		"chatType": "oneToOneChatting",
		"ownerUserId": "KHJDFhfs7dfgsvdfwsef",
		"chatWithUserId" "Afstg34tg4536gh"
	}
]

Now here you can see that I wont get any results, because I have Users in other collections but technically from App User perspective, he should see the chats:

  • Uysdf345GBHsdf
  • Fbgu283gbnap023656

Now in my mind I can do something like, I first search those users with searchText: User, and then collect all those _id from Users collection in array and match in Chatting collection, but this seems way too bad…

So can anyone tell me what is the best possible way to achieve such scenario?

2

Hey there,

If I understand your problem correctly it regards finding the correct data between multiple related collections. The best solution for this would probably be using aggregations. Here you can run an aggregation on the Users collection and use the $lookup operator to collect all chats related to the matched users. You might also be interested in a tool we have created at Vazco which greatly simplifies working with multi-collection queries: SparrowQL

If you could expand upon your problem specifically what kind of input do you have and what kind of output you would like to achieve I might be able to provide some more specific help.

3

Please check my updated question, I have added expected result for this scenario

4

Using aggregations are not compatible with subscriptions I assume.

5

@hassansardarlecodeur

I see the expected result, but I don’t I quite understand what is your input data. What information do you base your query on, Chatting.title and Users._id? And based on those two things you want to find all chats that a user can participate in?

As for subscriptions, do you mean Meteor subscriptions? If so, then there is no problem using them together. Aggregations are just like normal MongoDB queries.

6

I provided following search input:

user

So I am expecting following search results when I type user in search box:

[
	{
		"_id": "YONgsa793423bD",
		"title": "All users group",
		"chatType": "groupChatting"
	},
	{
		"_id": "Uysdf345GBHsdf",
		"chatType": "oneToOneChatting",
		"ownerUserId": "Afstg34tg4536gh",
		"chatWithUserId" "Xuibgsadbgsi35Gsdf"
	},
	{
		"_id": "Fbgu283gbnap023656",
		"chatType": "oneToOneChatting",
		"ownerUserId": "KHJDFhfs7dfgsvdfwsef",
		"chatWithUserId" "Afstg34tg4536gh"
	}
]
7

I think we’re missing something here? The search keyword alone is not enough criteria to find the other documents. Using regex to find this one document is okay:

	{
		"_id": "YONgsa793423bD",
		"title": "All users group",
		"chatType": "groupChatting"
	}

but the other two documents cannot be determined by this keyword alone as they don’t even have a title. Do you have some other input data like the user _id along with the keyword?

8

This was asked in Stackoverflow too and I also did not understand the question or the query or what exactly it is not working. From the information provided it is a bit like …look, I have green and red, I expect pink.

9

It is pretty clear if you study my question carefully, I am trying to search chats.

As you know in WhatsApp you have group chats and one to one chats on one single screen mixed up.

For group I have title, which I can search easily, but how can I search one to one chats as well?

Both groups and one to one chats are in same collection i.e Chatting, but for one to one chats I only store sender and receiverId in record

10

I think when multiple people are telling you they can’t understand your question, it’s clear that your question isn’t clear :).

That being said - are you trying to find all chats where either the title is “like” user or the name of the user involved is “like” user? If so, your options are:

  1. denormalize - track the names of the user involved in the chat in chatting - your best bet would be to make this an array users: [{ _id: "...", name: "..."}, { _id: "...", name: "..." }] - then you can query over either title or the name of the users - this is great when user names change infrequently (or never).
  2. As you suggest, look up users who match your query to get their user IDs then search for those - the problem here is reactivity, if you aren’t careful you’ll only be reactive on the chats of the found users, not the users who match the search.
2 Likes
11

Thanks a lot for getting my question. I think I am not that good with words :confused:

Actually I don’t think that your first point is worth trying, because the point where I will have too many chats then in that case if some users changes their names, its cost will be way too much.

And your second point is already something I am avoiding

12

Regarding cost of update in denormalization - it’s unusual for a user to change name, and the cost will scale not according to the number of chats globally, but the number of chats the specific users are involved in, the only potential problem I see is if you have thousands of users in a single chat. Of course you need to make sure you have appropriate indexes.

Your other option is to look into mongo aggregate - you will also have problems on reactivity here, but you could write a pipeline that performs the user search, then looks up on the users - the exact meteor and mongo versions you’re using will change the precise pipeline stages you use, but this is a third option

13

I think aggregate is also an overhead on db, so I assume I am left with option 1.
But what If I want to search messages, and the sender of message and those mentioned in a message?

14

For both messages and mentions - you’d do the same thing, attach the name of the mentioned and the name of the sender to the message, you’d then issue one query against chats and one against messages and merge the result

15

So for messages you want me to do something like this:

Messages:

[
	{
		"_id": "YONgsa793423bD",
		"senderId": "Xuibgsadbgsi35Gsdf",
		"senderName": "User A",
		"message": "Hello there!",
		"receiverId": "KHJDFhfs7dfgsvdfwsef",
		"receiverName": "User B",
	},
	{
		"_id": "YONgsa793423bD",
		"senderId": "KHJDFhfs7dfgsvdfwsef",
		"senderName": "User B",
		"message": "Hello @Xuibgsadbgsi35Gsdf",
		"receiverId": "Xuibgsadbgsi35Gsdf",
		"receiverName": "User A",
	}
]

So here I will be running an update query on messages on senderId, receiverId & mention ids, and update senderName, receiverName when ever user name is updated right?

16

It depends on a lot of things - if you only have sender/receiver then yes, this could work though you’d need 4 indexes to do it. A different option would be:

[
   {
      users: [{ _id: "", name: "", role: "sender|recipient|mentioned"}]
    }
]

then you have an index on users._id and users.name that way you could have an efficient query on users._id for updating and users.name for searching by name - though note that you can’t search by regex unless its case sensitive and starts with the phrase you’re looking for.

It seems like you’d benefit from reading some general information on schema design, indexing, query performance and optimizing for the most common cases. There’s some great information out there, but this forum is best for answering specific questions (e.g., “why doesn’t this specific query work”), rather than general “tell me how to design my schema” questions, as it’s hard for you to convey enough information to us that we can give you concise answers, and you end up in a situation where we have long discussions on pros/cons of all different approaches.

17

I wasn’t able to read the entire thread but from the first few topics, there was a question of search, then a suggestion of aggregation; and then about aggregation not being compatible with subscriptions. Not sure why there is a need for search and subscriptions.

Anyway, for our case with chat rooms, we sync our data to elasticsearch and used the collapse feature of elasticsearch which is good for these kinds of data relationships + search. Of course, there is also no subscription for our search. The actual chat rooms are still run by mongodb + meteor reactivity (only search is moved to elasticsearch)

18

Lets take a very simple example:

There is a collection where I store all my messages:

Messages:

[
	{
		"_id": "YONgsa793423bD",
		"groupId": "Phsdfyg92345sgb7651",
		"senderId": "Xuibgsadbgsi35Gsdf",
		"message": "hey there!"
	},
	{
		"_id": "sdgDFGbaofh135df",
		"groupId": "Phsdfyg92345sgb7651",
		"senderId": "KHJDFhfs7dfgsvdfwsef",
		"message": "Hello @Xuibgsadbgsi35Gsdf"
	}
]

In the above collection as you can I have sender Id & mentioned user id.

So my very basic question is that…

When I will search messages, I can search anything, by anything in messages means:

  • I can search anything in text message e.g: hello or there
  • I can search any user’s name to get all the messages which are being sent from specific user
  • I can search mentioned users aswell.

Now here the problem is that I am querying Messages collection, but users are placed in Users collection.

So I just want to know the best practice possible for this scenario.

I hope I have explain this properly this time

19

  • since Mongo@4.4 you can use the $unionWith operator in aggregation to search multiple collections altogether (best option for normalized data).

  • You can use external libraries to handle queries on related collections such as grapher from cult-of-coders.

  • You could also update one of the 2 collections and denormalize the data in one of the collection.

  • You can use the $graphLookup operator which is a $lookup operator with filter capacities.

  • You could externalize the search using an appropriate search engine such as Elastic Search (the best option for advanced text search).

1 Like
20

I think you probably just want to do two queries.

First, get the userIds of all the users that match your search query, like:

Users.find({ name: regex }, {name: 1}).map(user => user._id)

Then query Messages that are either written by these users OR match the search string:

Messages.find({
  $or: [{
    senderId: { $in: userIds }
  }, {
    message: regex 
  }]
})

You’ll want indexes on Users.name, Messages.senderId and Messages.messages. Depending on how big your collections are this may not scale too well, but at that point you would probably be better off looking into something like Elastic, as mentioned by someone above.

2 Likes