Ever since Monday morning my production app has been crashing randomly (and often). The last time I deployed any updates was approx a week before that. I have also rebuilt and restarted the app since then, but the crashes continue. Here is a stack trace from Kadira:
SOLVED: Apple license for push notifications expired
Which packages are you using, and do you interact with any other third party services (AWS S3 for example)
App is hosted on AWS EC2
Mongo is hosted on compose.com
I use Mixpanel for user analytics
I use Kadira for app monitoring
I use AWS S3 for user uploads (via the slingshot meteor package)
I use AWS Cloudfront for CDN
I use AWS SES for emails
Here are the meteor packages I use
accounts-base@1.4.2
accounts-oauth@1.1.15
accounts-password@1.5.0
alanning:roles@1.2.16
allow-deny@1.1.0
amplify@1.0.0
autoupdate@1.4.0
babel-compiler@7.0.6
babel-runtime@1.2.2
base64@1.0.11
binary-heap@1.0.10
blaze@2.3.2
blaze-html-templates@1.1.2
blaze-tools@1.0.10
boilerplate-generator@1.4.0
bozhao:link-accounts@1.2.9
browser-policy@1.1.0
browser-policy-common@1.0.11
browser-policy-content@1.1.0
browser-policy-framing@1.1.0
btafel:accounts-facebook-cordova@0.0.8
caching-compiler@1.1.11
caching-html-compiler@1.1.2
callback-hook@1.1.0
check@1.3.0
coffeescript@1.0.17
ddp@1.4.0
ddp-client@2.3.1
ddp-common@1.4.0
ddp-rate-limiter@1.0.7
ddp-server@2.1.2
deps@1.0.12
diff-sequence@1.1.0
dynamic-import@0.3.0
easysearch:core@2.2.0
ecmascript@0.10.6
ecmascript-runtime@0.5.0
ecmascript-runtime-client@0.6.2
ecmascript-runtime-server@0.5.0
edgee:slingshot@0.7.1
ejson@1.1.0
email@1.2.3
es5-shim@4.7.3
facebook-oauth@1.4.0
fourseven:scss@4.5.4
geojson-utils@1.0.10
hot-code-push@1.0.4
html-tools@1.0.11
htmljs@1.0.11
http@1.4.0
id-map@1.1.0
jquery@1.11.11
livedata@1.0.18
localstorage@1.2.0
logging@1.1.19
meteor@1.8.2
meteor-base@1.3.0
meteorhacks:kadira@2.30.5
meteorhacks:meteorx@1.4.1
meteorhacks:unblock@1.1.0
minifier-css@1.3.1
minifier-js@2.3.2
minimongo@1.4.3
modules@0.11.4
modules-runtime@0.9.2
mongo@1.4.3
mongo-dev-server@1.1.0
mongo-id@1.0.6
mongo-livedata@1.0.12
natestrauser:publish-performant-counts@0.1.2
npm-bcrypt@0.9.3
npm-mongo@2.2.34
oauth@1.2.1
oauth2@1.2.0
observe-sequence@1.0.16
ordered-dict@1.1.0
percolate:migrations@1.0.2
percolate:synced-cron@1.3.2
pindigo:analytics@1.0.0
pindigo:api@1.0.0
pindigo:collections@1.0.0
pindigo:mobile@1.0.0
pindigo:notifications@1.0.0
pindigo:uploads@1.0.0
promise@0.10.1
raix:eventemitter@0.1.3
raix:eventstate@0.0.4
raix:push@3.1.0-pre.1
random@1.1.0
rate-limit@1.0.9
reactive-dict@1.2.0
reactive-var@1.0.11
reload@1.2.0
retry@1.1.0
reywood:publish-composite@1.5.2
routepolicy@1.0.12
seba:minifiers-autoprefixer@1.0.1
server-render@0.3.0
service-configuration@1.0.11
session@1.1.7
sha@1.0.9
shell-server@0.3.1
shim-common@0.1.0
socket-stream-client@0.1.0
softwarerero:accounts-t9n@1.3.11
spacebars@1.0.15
spacebars-compiler@1.1.3
srp@1.0.10
standard-minifier-js@2.3.2
static-html@1.2.2
templating@1.3.2
templating-compiler@1.3.3
templating-runtime@1.3.2
templating-tools@1.1.2
tracker@1.1.3
u2622:persistent-session@0.4.4
ui@1.0.13
underscore@1.0.10
urigo:static-html-compiler@1.0.0
urigo:static-templates@0.1.3-custom
url@1.2.0
useraccounts-ionic-angular@0.0.1
useraccounts:core@1.14.2
useraccounts:ionic@1.14.3
webapp@1.5.0
webapp-hashing@1.0.9
here are the npm packages I use
"@babel/runtime": "7.0.0-beta.51",
"@meteorjs/ddp-graceful-shutdown": "0.9.2",
"angular": "1.6.10",
"angular-animate": "1.6.10",
"angular-elastic": "2.5.1",
"angular-meteor": "1.3.12",
"angular-moment": "1.2.0",
"angular-sanitize": "1.6.10",
"angular-ui-router": "0.4.3",
"bcrypt": "2.0.1",
"body-parser": "1.18.3",
"csv-writer": "1.0.0",
"highcharts": "6.0.4",
"highcharts-ng": "1.2.0",
"imgcache.js": "1.1.1",
"ionic-datepicker": "1.2.1",
"isomorphic-fetch": "2.2.1",
"lodash": "4.17.10",
"memobind": "0.5.0",
"meteor-node-stubs": "0.4.1",
"node-cache": "4.2.0",
"object-hash": "1.3.0",
"patreon": "0.3.0",
"pify": "3.0.0",
"probe-image-size": "4.0.0",
"query-string": "5.0.1",
"react": "16.4.1",
"react-bootstrap": "0.31.5",
"react-dom": "16.4.1",
"react-helmet": "5.2.0",
"react-loadable": "5.4.0",
"react-redux": "5.0.7",
"react-router": "4.3.1",
"react-router-config": "1.0.0-beta.4",
"react-router-dom": "4.3.1",
"redux": "4.0.0",
"redux-thunk": "2.3.0"
Your setup is quite similar to ours - we saw similar problems with the connection from our server to S3 (in the few cases where we have to communicate with S3 from our server) there is something odd about their node API - it throws errors which I could not catch in any way, I could log them, but they still crashed the entire server. We mitigated this by limiting our communication with S3.
I think its an outbound connection that is crashing, because as far as I know meteor exclusively uses HTTP, NOT HTTPS.
Some things to check:
Thanks for your time helping me think through this.
I’m checking Kadira to see what happened leading up to the crash, but haven’t figured it out yet.
It’s the notifications
aha, push notifications through which service? Android, AWS, Apple?
We send them for Android (firebase) and Apple using raix:push, but I’m not sure which one is crashing it
Talking to my partner who keeps track of keys and stuff to see if somethign expired.
Can you share your code for Android/firebase? I’ve been using it for a while without any problems, it could be you’re not handling an error somewhere. I’ve not used raix:push so can’t help you there - you know firebase can also handle IOS though?
I did not know it could handle both but that’s good to know. I let raix:push handle the notification setup for me, so I don’t have code to share unfortunately. I have a feeling I’m going to be re-implementing push notifications in my near future though. How did you go about that? What packages/plugins did you use?
Our Apple keys expired right around when we the server started crashing. Thanks for rubber ducking this one for me!
I just used request, but it will depend on how you’re using notifications - ours are always single device notifications
...
const msg = {
notification: {
title,
body: message,
icon: "/app/images/favicons/icon-128x128.png"
},
data: {
type: "success",
title,
body: message,
timeout: 10000
},
to: token.token
};
request.post({
url: "https://fcm.googleapis.com/fcm/send",
body: JSON.stringify(msg),
headers: {
Authorization: "key=REDACTED",
"Content-Type": "application/json"
}
}, Meteor.bindEnvironment((err, res, body) => {
try {
if (err) {
reject(err);
}
const jsonBody = JSON.parse(body);
if (jsonBody.success !== 1 || jsonBody.failure !== 0) {
Meteor.users.update({ _id: user._id }, { $pull: { deviceTokens: { token: token.token } } });
reject(new Error("bad token"));
}
else {
resolve();
}
}
catch (e) {
reject(e);
}
}));
...
And then you use the normal cordova plugin?
Kinda - we need to support browser notifications too, so there is some work around that. But on Cordova, yes - the way you submit a notification to firebase shouldnt impact the way you handle its receipt.