Sorry, need to clarify a bit more. We are talking about a couple hundred cards per user (which adds up to the hundreds of thousands of docs that @mexin was writing about). Actually, it’s mainly one collection (
Cards) which is displaying important information in the form of cards to the user. One clicking on these cards some specific actions are being triggered on the backend.
We do have 3 different pubs for this collection, due to the fact that there are different types of cards. The first two are restricted to 100 and respective 200 cards, the last one is unrestricted as there is a maximum of maybe 5-6 different ones that are displayed there.
The problem is that when we communicate with one of our partner sites via API we are creating a lot of cards, several per second. This goes on for about 3-4 minutes in total until all data is read and consumed and no more new cards are produced. So the problem is only happening on the initial load.
It seems that the observer has problems as the number of cards is constantly at that moment changing, handling the 3 queries (for each pub) and the conditions within (including the limits).
Cards collection. It’s likely that both are related.
A simple solution would be throttle the reactivity of pub/sub and only update new
Cards content every let’s say 250 or 500ms. But I have no idea how to do that.
The main problem it seems is that the GC has trouble with cleaning up behind and big memory leaks (from 1 to over 8 Mb’s) are created. This will lead eventually to a restart of our Galaxy backend server (we do have a different server for front- and backend).
Here are also two pictures that illustrate the memory leak problem.
First one shows the GC managing it’s cleanup until 1.5 minutes into the data loading:
The second one is showing that the memory leak in picture 1 is coming from the observer:
Hope this additional information helps, @mexin was writing this late at night his time.