Is there a de-facto solution for logging with Meteor, both client & server -side? Getting colored logs on file, rich logs to Mongo etc. would be really nice.
I spent an hour browsing Atmosphere & Github and these were my findings:
A quick search seems to have found me the creator of FiLog, @fgm . Could I trouble you for a short status update of your project? Searching for “FiLog” in Meteor Forums provided 0 results.
FiLog has only been published some days ago. Feel free to add issues on Github if you have any trouble using it. I’m specifically wondering how I should improve the quick start documentation : I feel the README.md might not be clear enough.
For its context, it has been developed for a pretty large application (around 20k LoC), to log both client and server events, locally in MongoDB for the short run and in ELK for longer term archiving. Still being worked on, so it’s likely to be a bit rough the first time someone outside the original team uses it, which is why I’m interested in feedback.
I’m using straight up bunyan npm package logger with ELK stack for storage. I really like the child logger capability that allows me to add context information for various loggers such as requestId, userId, etc and that will inherited by further child loggers to make it easy to trace something through the entire code flow.
I didn’t notice any specific load from callsite: but note that the goal is to only use this stack building for ‘error’-type events, so whatever load is entailed only applies when there is something to log. Is is not as if there was some sort of hook to go through.
Regarding positioning vs ELK / Loggly and others: the specifics of FiLog is that
it is really designed for Meteor, including a Meteor-specific processor as well as use of Meteor core on the server
it is compatible with ELK and this is actually one of the ways we use it: send logs to the local rsyslog/syslog-ng instance (to bypass the 1kB limit of standard syslog) and use FileBeat to ship to LogStash and thence to ElasticSearch, for use within Kibana
adding a Loggly sender would probably be a matter of just a few lines of wrapping code around the standard Loggly NPM module
Assuming you mean 1.4.2.1, since 1.4.5 is not yet a current Meteor version
In the main file of the server part of your app, create your logger instance like: let sender = new MongodbSender(Mongo, 'logger'); let logger = new ServerLogger( new LeveledStrategy(sender, sender, sender), WebApp); // And if you want to add user information to your logs logger.processors.push(new MeteorUserProcessor());
This is actually in the README.md, along with the associated client-side configuration.
In this simple example, your ServerLogger applies a logging strategy where all log levels use the same sender, which is a MongoDbServer, writing to your Meteor DB. Pushing the MeteorUserProcessor to the stack allows you to obtain Meteor accounts information for the current user.
Does FiLog have any kind of rate limiting implemented? I mean, if a client suddenly goes berserk and starts pushing 1000 log messages per sec, it might be an issue.
@arggh Not at this time, because the initial project didn’t need it, even on load tests. This is actually a hard problem: if you stop using TCP for UDP, you lose info ; if you keep it, you can swamp the main NodeJS/Meteor loop server-side with requests without any processing anyway. As I see it, for most actual cases, such rate limiting is a feature of the infrastructure, not of the applications. This is different from transation rate limiting, as in Meteor accounts-base, where the limitation is actually applied for a security scenario. For logs, you usually want to absorb as much as you can, to get as much information as possible.
If this turns out to really be an issue, it could be interesting to add something like the express-rate-limit module to the registration of the ServerLogger. Do you want to try it and submit a PR if it turns out well ?
@aadams To install, just do : meteor npm i filog --save
Then you must configure it into your application, by creating a logger instance to which you pass a strategy, to which you pass a number of senders, and you can push processors of your choice to adjust what you really want to store. Do this on client and server, each with its configuration. This is supposed to be explained in the README.md: if this not clear enough, could you suggest a new wording by sending a PR on github ?
I’ll first try to familiarize myself with this logger, but I’d be happy to contribute if necessary.
A few notes:
Could be worth mentioning, that the http-package is required
logger.arm() could be more pronounced in the docs
Might be a nice idea to include an example setup that’s configured for development / production with Meteor settings flags, for example
I think a guide on how to setup your own “logging server” would be nice as well, so that the clients don’t rely on the application servers for storing the logs
After running my app with the --production flag (locally), I got this in my server console every time the client tries to send a log entry:
Good remarks about what arm()/disarm() is for, and the http dependency. Do you mind adding these as an issue on the project ?
About “logging server”, you mean connecting with rsyslog/ELK, or something else ?
About --production, I suggest you open a bug report with your configuration (meteor version, and the client and server setups). Without more information, I have no idea how this can happen. Maybe because you haven’t passed all the needed environment variables on the command line (which I now realize are not documented in the README) ?
Maybe because you haven’t passed all the needed environment variables on the command line (which I now realize are not documented in the README) ?
@fgm: Could you elaborate on these env variables a bit? I added a couple of issues to the repo, also a reproduction for the issue regarding --production flag
With the “logging server” I meant the scenario, where I have … ~10 application servers running my app at app.mydomain.com, I would like them to report all log entries to a dedicated server at logger.mydomain.com. The sole task of this server is to catch these log entries, maybe put them in a separate database.
Probably it is as simple as just defining the URL when creating new MeteorClientHttpSender('http://logger.mydomain.com/logger) and some server config, right?
My reasoning behind this:
I don’t want any non-related processes to slow down my app servers, such as mailing, logging, batch processing etc…
My client might experience an error due to my server crashing or behaving badly, so I wouldn’t want to rely on that same server to log the incident. Better have a separate instance that catches the log entry
