App constantly refreshing after an update

help
61

thanks. good to know

62

Could this at all relevant to the problem?

Set HTTP_FORWARDED_COUNT to an integer representing the number of proxies in front of your server. For example, youā€™d set it to ā€œ1ā€ when your server was behind one proxy.
http://docs.meteor.com/api/connections.html#Meteor-onConnection

63

UPDATE: this post was wrong. Pretty sure the graph below happened for other reasons.

I think all connected clients may be inadvertently DDOSing the site now. After a recent update to all servers, despite far fewer clients connected, created observers has jumped massively

2240Ɨ390 33.9 KB

At the very least, is there a way I can stop the DDOS? (If this is the true cause of the massive jump in created observers)

64

Ouch @elie, I donā€™t envy youā€¦

Anyway, hereā€™s at bit more experience: We did a first beta (as in tests with real customers who have never seen the app before) test session this morning with a small group of people with very varied hardware and browsers. The site was running under 1.3.3-rc.4. Not a single reload loop occurred. I have not seen it at all since moving off 1.3.2.4.

This is still not proof that the issue is gone, but it kind of feels that way to meā€¦

1 Like
65

The only way that I could think of to stop these kinds of issues (too frequent requests from each client) is to deploy a new version, and use DDP Rate Limiter on each method call and publication.

66

Iā€™m not even sure itā€™s a DDP rate limiting issue. If the page is constantly refreshing, the CSS/JS of the page will be asked for constantly.

The worst bit about it is the user experience though.

For some reason it doesnā€™t constantly refresh for all users. But for me it has been doing so, and I know there are others too. Not at all good UX having to click the stop button and refresh the page to fix it.

@jesperwe was the upgrade to 1.3.3 beta smooth? Iā€™m going to try it tonight in any case

67

I donā€™t think DDP Rate limiter will help, the DDP frames look perfectly normal also when looping.

My upgrade was OK, I found #7184 which benjamn fixed really quickly, and one of my packages was using a NodeJS fs method that was deprecated and now was removed, but the maintainer fixed that quickly too, after that all was smooth. I guess it depends on what packages you useā€¦

68

So I upgraded to 1.3.3-rc4 beta this evening and it worked well. No more refreshing. Life saver! Thanks

2 Likes
69

Noooooo! The issue is back!! Was confident it had subsided! This is awful. No idea when the refreshing does or doesnā€™t happen. And Iā€™ve also seen it happen on the main domain, not just subdomains

70

Waitā€¦ my app has managed to go back to 1.3.2.4. Strangeā€¦

71

Just want to confirm that this issue does not go away after upgrading to Meteor 1.3.3-rc5.

72

Apologies for talking to myself here, but I have a strong feeling this bug is killing my servers a little.

Hereā€™s the latest update in the story:
Some of my servers are hitting 100%. I couldnā€™t understand why.
Here are two Kadira graphs from one of my instances:

Graph 1:

2380Ɨ370 40.5 KB

Graph 2:

1168Ɨ812 40.1 KB

Interesting correlation between the graphs.

Whatā€™s more interesting is that I remove the raix:push package a few days ago. The wait time you see in Graph 1 is an throwing errors for not finding the method raix:push-setuser.

How is this method being called if I removed it from the app? The answer seems to be that there are still clients calling old Meteor code. And Iā€™m guessing this is the infinite refresh problem some of us have been having. There are certain browsers that are connected to my app that have been refreshing for days straight and during all these refreshes they seem to be running old code and bombarding my servers to fetch data from the database causing 100% CPU.

So it looks like it keeps reloading old code, sees the code is old and then reloads again. Or something along those lines. The problem is the new code never comes in.

@abernix: does this make sense? And any ideas how we can get this issue fixed?

73

Youā€™re not talking to yourself, but as I have not seen the problem since moving to 1.3.3 it is even harder get useful info.

My reasoning on all this is about the following:

Known Facts

  • Only a few people are seeing this, and always intermittently
  • We have not been able to establish many common parameters between our environments:
  • Different deployments (single/multiple servers, mupx / not mupx etc)
  • Different routers
  • Intersection of packages used between us is basically zero.
  • Everyone runs nginx proxy, but polishing the nginx config does not help.
  • Analysis of the WS frames show nothing suspicious.
  • Reloads happens with DISABLE_WEBSOCKETS=true as well.
  • The reload-safetybelt package does not help.
  • The reload happends after a significant portion of the app has initialized, even after some subscriptions have been fulfilled etc.
  • Once the preconditions leading to a reload loop have been fulfilled, they stay so for that client (i e the looping never stops)

ā€¦soā€¦what gives?

Somewhere in Meteor there must be code that does a reload. If not, this is a weird browser bug??
This code gets triggered by some unusual combination of events, most probably involving some sort of race condition giving us the intermittent behavior, and also involving some dependency on something getting cached in a proxy.

To me, to debug this further you need either to be an expert on Meteor internals, or you need a repeatable testcase, which we have not :worried:

74

@elie, after reading my post above a few time, one thing that stands out is this:

  • The issue is intermittent as to when it STARTS TO HAPPEN
  • Once started, it is NO LONGER intermittent, it happens repeatably on every reload.

This means that during looping, something is stuck somewhere (=in the browser or in the proxy) for that client.

Could it be this is an nginx issue? I know youā€™re desperate. Would it be worth the effort for you to try a different reverse proxy package?

75

I assume it could well be an nginx config issue. Iā€™d rather not update the
app than having to move away from nginx myself.

I started to have a look at the meteor internals. The reload package is
almost certainly where the reloading is happening.

The issue does seem to be happening less to me at the moment, but that also
makes it harder to debug.

I will continue to investigate and I think I might use the dev version of
meteor or edit it and add some console logs to figure out whatā€™s happening.

76

Hello
we had the same issue and it belongs to the new caching mechanism of Meteor 1.3.x. You have to disable caching in your nginx file and the refreshing will be gone. Let the caching happening in Meteor itself.
# pass all requests to Meteor location / { proxy_pass http://yourapp:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; # allow websockets proxy_set_header Connection $connection_upgrade; proxy_set_header X-Forwarded-For $remote_addr; # preserve client IP }

77

Where have you disabled caching? This looks like the standard nginx Meteor conf file? Has something been removed from it?

Should the section at the bottom of this file be removed?

# pass all requests to Meteor
location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade; # allow websockets
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header X-Forwarded-For $remote_addr; # preserve client IP

    # this setting allows the browser to cache the application in a way compatible with Meteor
    # on every applicaiton update the name of CSS and JS file is different, so they can be cache infinitely (here: 30 days)
    # the root path (/) MUST NOT be cached
    if ($uri != '/') {
        expires 30d;
    }
}

Also, any idea where the Meteor caching code is located in the Meteor code? (Is it somewhere in here? https://github.com/meteor/meteor/blob/devel/packages/webapp/webapp_server.js)

And why canā€™t nginx handle the caching everything apart from / since 1.3?

I will try removing the caching this evening and see what happens.

78

I had this same issue, which only started after I added meteorhacks:cluster and started running two processes. Once I removed meteorhacks:cluster and returned to a single process the constant reloading was gone as well.

79

Does anyone have an idea of how to just stop clients running the old code? There are browsers around the world constantly refreshing an old app and Iā€™d like it to stop.

Iā€™m not sure whether remove expire from nginx has helped anything. Itā€™s difficult to know if the problem is resolved since it doesnā€™t always happen.

80

4 weeks on and I donā€™t think weā€™re any closer to a solution. I donā€™t think any of the solutions in this thread have helped. Although they have apparently helped in individual cases.

The problem still exists in meteor 1.3.3