I’ve recently noticed a high number of Sessions on Meteor APM which didn’t match my Analytics live user count. My app is deployed on 5 servers. The first one (which hosts the load balancer) reports about 500 open sessions while other four have 10 each, and Analytics reports 50 current users.
I looked at IPs and User-Agent to eliminate the possibly of bots/DDoS and noticed some IPs open multiple DDP connections in a very short time frame. Based on IP and User Agent pattern, these look like legit users. I also added Cloudflare DDoS protection just in case.
I can’t figure out what might be causing this super high number of connections. App seems to be working just fine (as long as the servers take the load) and I can’t reproduce these multiple connections locally. My app isn’t doing anything particularly fancy nor do I have other apps/clients using DDP.connect. APM also isn’t showing any suspicious Method or Subscription.
Any idea on what might be causing these multiple simultaneous connections? Any suggestion on how to further investigate?
The # of sessions i’m referring to is the one measured by Monti APM (Kadira). Purple is server 1, others are below.
I’ve logged new connections by using Meteor.onConnection(callback). I’ve sometimes noticed ~10 connections within 1sec. from the same IP but I don’t believe each user is opening 10 tabs.
My app was deployed on Heroku with multiple dynos when I noticed this. I’ve now moved it to AWS with Mup deployment (loadBalancing + sticky sessions) and seeing no difference.
Do you see any errors that would indicate connection problems? Similarly, are all the sessions active? E.g., do you see traffic on each of the sessions? Do the logins from the same ip all eventually login with the same user (assuming you have logins). Just kinda spitballing here, I have seen users open a stupid number of tabs in the past, particularly if your application is designed to open multiple tabs.
There was indeed a problem with my Nginx config. I was using Cloudflare as a proxy and hadn’t configured Nginx to get real IPs so the load balancer was only seeing Cloudflare IPs (for anybody interested: Getting Real IP Addresses Using CloudFlare, Nginx, and Varnish - Daniel Miessler). After fixing this, connections are properly distributed on the servers.
That being said, the number of connections is still about 10x the number of unique users (based on GA + Cloudflare stats). Looking at Nginx logs I can see that most requests are for /socks/info (are these just normal WS messages passing through?) and come from Windows 10/Chrome 89 users (could just be the most common setup…).
/socks/info controls whether a websocket (if supported by the client). or HTTP requests are used as well as some other config.
It shouldn’t be called frequently unless there are connectivity problems (this can be caused by sessions timing out by a firewall, or just regular network problems. When I open a meteor site I see one request to /socks/info followed by one request to /socks/*/*/websocket. If your clients don’t support websockets for some reason, or if cloudflare/your nginx setup are blocking websockets that could be the problem too
