Proper logging with Meteor

Is there a de-facto solution for logging with Meteor, both client & server -side? Getting colored logs on file, rich logs to Mongo etc. would be really nice.

I spent an hour browsing Atmosphere & Github and these were my findings:

Any first hand experience doing proper logging with Meteor?

5 Likes

A quick search seems to have found me the creator of FiLog, @fgm . Could I trouble you for a short status update of your project? Searching for “FiLog” in Meteor Forums provided 0 results.

Found a few more:

1 Like

FiLog has only been published some days ago. Feel free to add issues on Github if you have any trouble using it. I’m specifically wondering how I should improve the quick start documentation : I feel the README.md might not be clear enough.

For its context, it has been developed for a pretty large application (around 20k LoC), to log both client and server events, locally in MongoDB for the short run and in ELK for longer term archiving. Still being worked on, so it’s likely to be a bit rough the first time someone outside the original team uses it, which is why I’m interested in feedback.

I kind of like Loggly. They have a guide for Meteor:
https://www.loggly.com/blog/managing-a-meteor-application-in-production-three-real-log-management-use-cases/

Big bonus, it starts free.

I also like Kadira error capturing. Does both server & client.
Also starts for free.

2 Likes

is the performance of callsite such that you can keep it on in production in a busy environment?

I’m using straight up bunyan npm package logger with ELK stack for storage. I really like the child logger capability that allows me to add context information for various loggers such as requestId, userId, etc and that will inherited by further child loggers to make it easy to trace something through the entire code flow.

I didn’t notice any specific load from callsite: but note that the goal is to only use this stack building for ‘error’-type events, so whatever load is entailed only applies when there is something to log. Is is not as if there was some sort of hook to go through.

Regarding positioning vs ELK / Loggly and others: the specifics of FiLog is that

  • it is really designed for Meteor, including a Meteor-specific processor as well as use of Meteor core on the server
  • it is compatible with ELK and this is actually one of the ways we use it: send logs to the local rsyslog/syslog-ng instance (to bypass the 1kB limit of standard syslog) and use FileBeat to ship to LogStash and thence to ElasticSearch, for use within Kibana
  • adding a Loggly sender would probably be a matter of just a few lines of wrapping code around the standard Loggly NPM module

I think I’ll just stop wasting time pondering on the options and bolt FiLog to my app. Let’s see how it goes :slight_smile:

@fgm I understood you are using it in a production app already without any major issues?

Can you add FiLog to a Meteor 1.3.5.1 app and if so how do you install it? Also, how do you point the output to my MongoDB?

Not yet: it’s currently in preprod for end-user (it’s an intranet app) testing.

Assuming you mean 1.4.2.1, since 1.4.5 is not yet a current Meteor version :slight_smile:

In the main file of the server part of your app, create your logger instance like:
let sender = new MongodbSender(Mongo, 'logger'); let logger = new ServerLogger( new LeveledStrategy(sender, sender, sender), WebApp); // And if you want to add user information to your logs logger.processors.push(new MeteorUserProcessor());
This is actually in the README.md, along with the associated client-side configuration.

In this simple example, your ServerLogger applies a logging strategy where all log levels use the same sender, which is a MongoDbServer, writing to your Meteor DB. Pushing the MeteorUserProcessor to the stack allows you to obtain Meteor accounts information for the current user.

Sorry, I mean Meteor 1.3.5.1. Also, I mean how do you install it?

Does FiLog have any kind of rate limiting implemented? I mean, if a client suddenly goes berserk and starts pushing 1000 log messages per sec, it might be an issue.

@arggh Not at this time, because the initial project didn’t need it, even on load tests. This is actually a hard problem: if you stop using TCP for UDP, you lose info ; if you keep it, you can swamp the main NodeJS/Meteor loop server-side with requests without any processing anyway. As I see it, for most actual cases, such rate limiting is a feature of the infrastructure, not of the applications. This is different from transation rate limiting, as in Meteor accounts-base, where the limitation is actually applied for a security scenario. For logs, you usually want to absorb as much as you can, to get as much information as possible.

If this turns out to really be an issue, it could be interesting to add something like the express-rate-limit module to the registration of the ServerLogger. Do you want to try it and submit a PR if it turns out well ?

@aadams To install, just do : meteor npm i filog --save

Then you must configure it into your application, by creating a logger instance to which you pass a strategy, to which you pass a number of senders, and you can push processors of your choice to adjust what you really want to store. Do this on client and server, each with its configuration. This is supposed to be explained in the README.md: if this not clear enough, could you suggest a new wording by sending a PR on github ?

I’ll first try to familiarize myself with this logger, but I’d be happy to contribute if necessary.

A few notes:

  • Could be worth mentioning, that the http-package is required
  • logger.arm() could be more pronounced in the docs
  • Might be a nice idea to include an example setup that’s configured for development / production with Meteor settings flags, for example
  • I think a guide on how to setup your own “logging server” would be nice as well, so that the clients don’t rely on the application servers for storing the logs
  • After running my app with the --production flag (locally), I got this in my server console every time the client tries to send a log entry:
I20161111-11:08:00.776(2)? [SyntaxError: Unexpected end of input]
I20161111-11:08:00.777(2)? [SyntaxError: Unexpected token e]
I20161111-11:08:00.777(2)? [SyntaxError: Unexpected token (]
I20161111-11:08:00.779(2)? [SyntaxError: Unexpected token p]
I20161111-11:08:00.779(2)? [SyntaxError: Unexpected token s]
I20161111-11:08:00.779(2)? [SyntaxError: Unexpected token p]
I20161111-11:08:00.782(2)? [SyntaxError: Unexpected token \]
I20161111-11:08:00.783(2)? [SyntaxError: Unexpected token g]
I20161111-11:08:00.786(2)? [SyntaxError: Unexpected token :]
I20161111-11:08:00.787(2)? [SyntaxError: Unexpected token (]
I20161111-11:08:00.788(2)? [SyntaxError: Unexpected token o]
I20161111-11:08:00.788(2)? [SyntaxError: Unexpected token s]
I20161111-11:08:00.789(2)? [SyntaxError: Unexpected token ä]
I20161111-11:08:00.790(2)? [SyntaxError: Unexpected token b]

Any idea about that one?

I’ll keep playing :slight_smile:

  • Good remarks about what arm()/disarm() is for, and the http dependency. Do you mind adding these as an issue on the project ?
  • About “logging server”, you mean connecting with rsyslog/ELK, or something else ?
  • About --production, I suggest you open a bug report with your configuration (meteor version, and the client and server setups). Without more information, I have no idea how this can happen. Maybe because you haven’t passed all the needed environment variables on the command line (which I now realize are not documented in the README) ?

Maybe because you haven’t passed all the needed environment variables on the command line (which I now realize are not documented in the README) ?

@fgm: Could you elaborate on these env variables a bit? I added a couple of issues to the repo, also a reproduction for the issue regarding --production flag

With the “logging server” I meant the scenario, where I have … ~10 application servers running my app at app.mydomain.com, I would like them to report all log entries to a dedicated server at logger.mydomain.com. The sole task of this server is to catch these log entries, maybe put them in a separate database.

Probably it is as simple as just defining the URL when creating new MeteorClientHttpSender('http://logger.mydomain.com/logger) and some server config, right?

My reasoning behind this:

  1. I don’t want any non-related processes to slow down my app servers, such as mailing, logging, batch processing etc…
  2. My client might experience an error due to my server crashing or behaving badly, so I wouldn’t want to rely on that same server to log the incident. Better have a separate instance that catches the log entry

FTR, the --production issue fix has been merged yesterday, thanks for reporting the issue.

1 Like