Chat Performance Optimization

hayk · February 25, 2018, 3:42pm

So recently I wrote a Chat application. The nuance with this app is that there is a user in the actual meteor app on one hand and on the other hand there is another user in some non-meteor app enviroment using a ddp client (currently this one https://github.com/seeekr/ddp-client).
On both sides for a message to arrive takes around 4 seconds.
Where from should I start optimizing this?
I never did optimization so I am not sure what to do.
Can you at least provide me with some guidelines for this situation so that I know what are my steps?

copleykj · February 25, 2018, 5:17pm

Can you tell us more about the app, it’s a architecture and how concurrent users it gets? Do you have oplog enabled?

Chat will be notoriously hard on Meteor’s live data system, but 4 seconds seems pretty high. At the very least you’ll want to make sure your collections are properly indexed. To mitigate the issues with live data, you may want to consider using redis-oplog as well.

hayk · February 26, 2018, 6:39am

Sure.
I am not sure I have enough knowledge about oplog and redis-oplog however reading this

Just add RedisOplog, you will already see big performance improvements

So I’ll try adding this package, now.

As for the structure

I have 2 apps connected to the same db.
One is the main app, our users use. And the other one is for handling the connections from outside the app.

Here is a sample document from Chats collection

{
	"_id" : "immAqcR7zY2PpMpPi",
	"contactId" : "bLZrqu25jN94QQJ3Q",
	"projectId" : "LkJnFLPKTRwCAfNem",
	"lastVisitSessionId" : "vlcwegLH4ra1TvyCByhp5Yi12kkG1BCl",
	"createdAt" : ISODate("2018-02-25T15:00:25.785Z"),
	"messages" : [
		{
			"visitSessionId" : "vlcwegLH4ra1TvyCByhp5Yi12kkG1BCl",
			"createdAt" : ISODate("2018-02-25T15:00:25.785Z"),
			"userId" : "kGgnMxXyp7paSvgLk", // if exists, message is from the user
			"message" : "Welcome dear visitor XXX"
		},
		{
			"visitSessionId" : "vlcwegLH4ra1TvyCByhp5Yi12kkG1BCl",
			"createdAt" : ISODate("2018-02-25T15:00:42.803Z"),
			"userId" : null, // if null message is from the contact
			"message" : "xxx"
		},
  ]
}

This is how I publish it

Meteor.publish('Chats', function ({contactId}) {
  return Chats.find({contactId})
})

And methods from the second app

'chat.init' ({ email, convertedPage, projectId, visitSessionId }) {
    console.log('chat.init', { email, convertedPage, projectId, visitSessionId })
    if (!projectId) throw new Meteor.Error('invalidEmail', 'Please enter a valid email')
    if (!isValidEmail(email)) throw new Meteor.Error('invalidEmail', 'Please enter a valid email')
    const contact = Contacts.findOne({email, projectId})
    let contactId = ''
    let chatId = ''
    const settings = ChatSettings.findOne({projectId})
    const newChat = {
      contactId,
      projectId,
      lastVisitSessionId: visitSessionId,
      createdAt: new Date(),
      messages: [
        {
          visitSessionId,
          createdAt: new Date(),
          userId: Projects.findOne(projectId).ownerId, // TODO: think about generalAccount
          message: settings.defaultMessages.welcome
        }
      ]
    }
    if (contact) {
      contactId = contact._id
      newChat.contactId = contactId
      const chat = Chats.findOne({contactId})
      if (chat) chatId = chat._id
      else chatId = Chats.insert(newChat)
    } else {
      contactId = Contacts.insert({email, visitSessionId, projectId})
      newChat.contactId = contactId
      chatId = Chats.insert(newChat)
    }
    return { contactId, chatId }
  },
  'chat.submitContactMessage' ({ visitSessionId, contactId, message }) {
    if (!contactId) throw new Meteor.Error('noContact', 'Please submit your email')
    if (!message) throw new Meteor.Error('emptyMessage', 'Please enter a message')
    return Chats.update({contactId}, {$push: { messages: {
      visitSessionId,
      createdAt: new Date(),
      userId: null,
      message
    }}})
  },

And a method from the main app

'chat.submitMessage' ({ _id, message }) {
    console.log('chat.submitMessage', message)
    if (!message) throw new Meteor.Error('emptyMessage', 'Please enter a message')
    const chat = Chats.findOne(_id)
    return Chats.update({_id}, {$push: { messages: {
      visitSessionId: chat.lastVisitSessionId,
      createdAt: new Date(),
      userId: Meteor.userId(),
      message
    }}})
  },

Here are some indexes

db.Chats.getIndexes()
[
	{
		"v" : 2,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "btctestdb.Chats"
	},
	{
		"v" : 2,
		"key" : {
			"contactId" : 1
		},
		"name" : "contactId_1",
		"ns" : "btctestdb.Chats",
		"background" : true
	}
]

db.contacts.getIndexes()
[
	{
		"v" : 2,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "btctestdb.contacts"
	},
	{
		"v" : 2,
		"key" : {
			"projectId" : 1
		},
		"name" : "projectId_1",
		"ns" : "btctestdb.contacts",
		"background" : true
	},
	{
		"v" : 2,
		"key" : {
			"email" : 1
		},
		"name" : "email_1",
		"ns" : "btctestdb.contacts",
		"background" : true
	},
	{
		"v" : 2,
		"key" : {
			"email" : 1,
			"projectId" : 1
		},
		"name" : "email_1_projectId_1",
		"ns" : "btctestdb.contacts",
		"background" : true
	}
]

copleykj · February 26, 2018, 2:41pm

Thanks for the additional info. In this case neither use of oplog or redis-oplog are going to be able to fix the problem because the issue lies in the structure of your data. To fix the issue you’ll need to normalize your data into other collections. Specifically your messages. I believe that currently every time a message is added to the array, that the entire array will have to be sent to the client again. Also if you are using blaze the rendering will be much less efficient when iterating over an array instead of a mongo cursor so that could affect performance in the UI as well. Another performance implications of your data structure is going to be the fact that you can not paginate your messages and will have to publish all of them any time you want even one of them, and as that array grows, so will your bandwidth and performance issues.

If you need some ideas as how to structure you data, you can take a look at my socialize:messaging package.

hayk · February 26, 2018, 2:49pm

Thanks!
I was guessing the issue would be with the data structure but I couldn’t exactly point my finger on that!
You helped me a lot! I am sure the issue will be fixed after fixing the structure.

diaconutheodor · February 26, 2018, 2:56pm

@hayk if you want relational help also look at Grapher: https://github.com/cult-of-coders/grapher

hayk · March 2, 2018, 1:41pm

Yes, that’s something I am looking to implement. But if I can find a quicker solution for now that would be better.

hayk · March 2, 2018, 1:51pm

@copleykj
So I did some restructuring. It turned out Chats collection is unneccassary for me at all. I just need messages like these

{
  "_id": "TLjzhBJFHDCPnYapW",
  "contactId": "Lh9kSCFjhGyY9aqW3",
  "projectId": "LkJnFLPKTRwCAfNem",
  "visitSessionId": "q6ibdw3eSw5qjXGbhnRPOCPWsS7f9jDk",
  "createdAt": ISODate("2018-03-02T13:02:00.952Z"),
  "userId": null,
  "message": "Hello",
  "isRead": true
}

I just publish them either by contactId or by projectId
So I added separate indexes for these fields.
However there weren’t any performance boosts. So I also added redis-oplog to both of my apps.
Now a very interesting thing happens.

I open 2 browser windows. So I send the message in one window. In the next window the message arrives as usual with around 4 seconds delay. BUT if I send the message and click on the other window, it appears instantly. Do you have any idea why this might happen?

copleykj · March 2, 2018, 2:07pm

That is a very strange issue, especially since it sounds like it’s happening locally. Honestly I don’t think I would have much chance of figuring this out without digging through your application code.