The new APM for Meteor is coming soon!

announce
1

meteor-elastic-apm

Performance Monitoring for Meteor based on Elastic APM

what it monitors

  1. Meteor methods execution - it tracks their execution time with detailed information of what db queries was executed
  2. Meteor pub/sub, tracks publications response time
  3. Meteor pub/sub - operations, how much documents was added, updated, removed
  4. Async ops inside methods and pubs, for example http requests
  5. Incoming and outcoming HTTP requests
  6. Errors - with detailed information and stack traces

what it doesn’t monitor

  1. CPU usage
  2. Memory usage
  3. Session count

more screenshots

project status - RC

getting started

  1. Install and configure elasticsearch - https://www.elastic.co/downloads/elasticsearch
  2. Install and configure Kibana - https://www.elastic.co/downloads/kibana
  3. Install and configure elastic APM server - https://www.elastic.co/downloads/apm

Then

meteor add kschingiz:meteor-elastic-apm

Then somewhere in your server code, Elastic documentation stays that Agent.start should be executed before anything else, and should be at the very top of your code

import Agent from 'meteor/kschingiz:meteor-elastic-apm';
Agent.start(options);

Kibana APM with Meteor with MUP

Meteor Up is a production quality Meteor app deployment tool. We expect you already has up and running Meteor app on server deployed with MUP.

  1. mup ssh
  2. wget https://raw.githubusercontent.com/elastic/apm-server/master/apm-server.yml && cp apm-server.yml /etc/apm-server/apm-server.yml
  3. Now you need to edit /etc/apm-server/apm-server.yml, at least you need to add you elastic search url under output.elasticsearch. When you finish just close this terminal
  4. Now we need to update mup.js file to:
    a) Install apm-server in app container
    b) Pass apm-server config file into our app container
    c) Start it everytime after deploy
{
  app: {
    ...
    volumes: {
      '/etc/apm-server/apm-server.yml': '/etc/apm-server/apm-server.yml'
    },
    docker: {
        ...
        buildInstructions: [
        // https://www.elastic.co/guide/en/apm/server/current/setup-repositories.html
        'RUN apt-get install wget -y',
        'RUN wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -',
        'RUN apt-get install apt-transport-https',
        'RUN echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list',
        'RUN apt-get update && apt-get install apm-server -y',
        'RUN update-rc.d apm-server defaults 95 10'
        ]
    }
    ...
  },
  ...
  hooks: {
    // Run apm-server
    'post.deploy'(api) {
      return api.runSSHCommand(
        api.getConfig().servers.one,
        'docker exec development service apm-server start'
      );
    }
  },
}

demo app

API

Agent is based on elastic/apm-agent-nodejs and fully supports all of it’s features https://github.com/elastic/apm-agent-nodejs

contributions

All contributions are welcome, Let’s make the better APM together!

27 Likes
2

Glad to see another option! Good luck on making this work.

Before I even dive into it, I would recommend that agent start should be automatic if the package is included. In the same way that Kadira makes that currently.

3

Thanks you for this idea, I am planning to do this after I complete the main part (features, bug fixes, etc.)

4

Good stuff. A few questions:

  1. Have you thought about using meteor-apm as the base package? I believe MDG has solved some bugs that were present in Kadira.
  2. Have you thought about just creating a patch for kadira-client/meteor-apm to push the data they output directly into a database?
  3. To follow-up on question 2, Mongo support would be nice and easy, and then one could swap in for Elasticsearch as volume goes up
5

Hi, thanks for your good questions:

  1. Yes, I am using official meteor-apm-client as a template, because it has +20 commites with bug fixes.
  2. I got your point, you mean Kadira server writing directly to elastic? No, I think it will be too complicated, and the main reason why I am developing this package is to throw out Kadira, so Kadira server will not be used anyway.
  3. No, because Elastic APM has some specific features that Kadira doesn’t support. We can’t easily switch between these APMs.

Now I am working to fully copy the main Kadira functions and metrics, but the development is going slow. Soo please be patient I will release the first production version as soon as possible.

6

Hmm - but the data is being generated on the server where the application runs, doesn’t it? I thought it can be easy to just store it in a database and then build some kind of UI to visualize it.

7

That’s great that you’re working on that @kschingiz. You should also check out Monti APM which is being actively developed.

One thing that would be really cool to work into your design of your agent from the start would be the idea of different providers. So maybe the existing Kadira server could be one, your new solution with Elastic could be another and then others could come along and add theirs in the future. It would be really useful if it could send different data to different providers too, so maybe client errors to Sentry and server errors and performance metrics to Elastic.

1 Like
8

@marklynch That’s a great idea, I was doing something like you mean in other repository
It supports multiple senders, can be easily extended to support new one, my example was with graphite apm
But the project was abandoned, because I want to complete with elastic apm.
Maybe I will do something you said in future versions.

That’s great that we have MontiAPM based on Kadira, but I want these kind of tools in open source world.

1 Like
9

I don’t understand you, you mean to build some new custom UI that will visualize the saved data? Why do we need this? It will be easier to take some open source APM server and integrate it with meteor app.

10

Great to see alternatives rising up. Will definitely follow your work. Since the ELK stack is a quite common setup I like the idea of building on top of those tools!

11

I have added new features:

  1. APM monitors all incoming http requests, useful if you have REST
  2. Bug fixes

TODO:

  1. Monitor all outcoming http requests
  2. Track user information
  3. Demo application which show all of APM features in detail
  4. Documentation
  5. Tutorials
    After that I think the first production ready version will be released.
2 Likes
12

Thanks a lot for your work put down here.

I would very much like to replace my Kadira installation with the elastic-stack just because it’s monitoring support seems to be so much more flexible when it comes to other things than Meteor itself.

I know you’ve listed what this package monitors, but could you also provide a list (or a todo-list) of features Kadira has but this stack does not support as of now?

Btw. I like the fact you added http-request monitoring. Something which is a reason for me to switch right away.

13

Thanks a lot for your feedback. I am now testing this package in my test projects, when I will be sure that it’s production ready I will release it with detailed documentation and comparison with Kadira.
For now I can say that:

  1. Kadira has more convenient oplog operations graph than elastic apm
  2. Kadira CAN count CPU, Sessions and Memory usage, my package can’t
  3. Kadira CAN NOT monitor all outcoming requests, only which are done using HTTP.call. My package monitors ALL outcoming requets
  4. Kadira CAN catch client side exceptions, Elastic-apm only server side (TO-DO)
  5. Elastic-apm CAN NOT show detailed information in spans, f.e. I can only record that some document has been inserted into MongoDB but APM can’t show it’s properties. (Elastic says that in future they will fix this)
  6. Kadira crashes when you have +5GB database size (my case, I have to clear db every day), Elastic can handle easily much more data than Kadira.
  7. Elastic-APM can’t monitor outcoming emals, I am now thinking how to do it more generic as I did with outcoming HTTP

I can’t do 1. and 2. because I don’t know yet how to show transactions in convenient time-series graph (Not a priority now, I will ask Elastic how to do this in future)
I will be happy if you help with testing and give me a feedback of how it works with your project. As I said before: Let’s make better APM together.
Thanks!

2 Likes
14

Just to add a cross reference … You’ve maybe already heard of it: Kadira improvements

1 Like
15

Yes, I have seen this topic. Thanks for the reference. I will learn the source code and adapt its improvements to my package later. Now I am thinking just to release the production ready package

16

Thank you for your work, I’ll follow it!

1 Like
17

I just added how to install apm-server with meteor up, I’ll add later full tutorial to install ELK stake and connect meteor APM & Logs

18

That’s great :+1: Thanks for your help! I think we are ready to release it as a production ready, because I didn’t have any exceptions or errors in my test projects. Tomorrow I will publish test projects if you want to try it.

1 Like
19

Hi there! I have developed the very simple demo application:


You are welcome to try and test it.
Thanks!

2 Likes
20

Just FYI I had written a NewRelic package for Meteor and we’re poised to switch to Elastic APM due to the NR subscription costs, so it’s likely I’ll be able to help soon on this project.

2 Likes