SSR memory leaks

Okay something else weird - it only leaks in production. If I put the app up on Staging (and then hammer it, with more traffic than we get on the main site) it doesn’t leak memory. The only difference between staging and production is the database they each connect to (both on Atlas, both Replica Sets with 3 nodes, on Version 4.0.12 - staging is a “sandbox”), and on production we use a “professional” container to get access to APM - but the memory leaks and restarts were a problem before that.

Obviously, the production database is much larger (Atlas says 1.8Gig, which seems larger than it should be), and the number of connections - I get 30 on production, and only 4 on staging. I have no idea what that means - I guess I have my reading cut out for me.

We experienced something similar before wherein if a link of a page is posted in a facebook page, a handful of facebook bots plus traffic can kill an aws ec2 instance before it can spawn due to memory use.

We did not think it was a memory leak that time but we end up caching our ssr pages through redis. We are using your react-loadable fork for ssr

I guess it could be bots. This particular site gets almost no traffic according to the connections count in Galaxy (there are literally 0 right now). I’ll try sharing from staging and see what happens.

I’m also going to try heap save and see what that reveals.

Thanks for the tips all!

Another thing I just noticed - The CPU when the server is leaking memory spikes up to 60% every 30 seconds on the production server. This doesn’t happen on the staging server.

Did you see anything in the APM accessing or executing every 30 seconds?

Yes! It only does that on production though. I put the same code connected to the same database on staging (another galaxy site), and no leak there. I put in a support ticket last night, but no response yet.

Oh, do you mean a specific function? I don’t know how to check that - APM doesn’t seem as useful as I remember back in the day. Maybe I’m just not finding the right buttons.

I had this issue once. Had to replace one for loop by map function. Then solve the issue. I really don’t know why but it works. Haha.

It looks like it can handle limited of async/await levels.

I wonder if this can be causing my leaks. https://github.com/facebook/react/issues/13854

I’m going to try [this package(https://github.com/panter/meteor-fiber-save-react-context), which seems to patch for that.

1 Like

I have that problem too. Still looking for a solution. Thank you.

It didn’t work, still leaking. But I think that issue may have something to do with it. Right before I run out of memory, I get this message:

2019-10-03 21:15:56-04:00Error: Minified React error #321; visit https://reactjs.org/docs/error-decoder.html?invariant=321 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
2019-10-03 21:15:56-04:00 at W (/app/bundle/programs/server/npm/node_modules/react/cjs/react.production.min.js:20:386)
2019-10-03 21:15:56-04:00 at useContext (/app/bundle/programs/server/npm/node_modules/react/cjs/react.production.min.js:22:416)
2019-10-03 21:15:56-04:00 at Loadable (packages/npdev:react-loadable/react-loadable-server.js:48:21)
2019-10-03 21:15:56-04:00 at d (/app/bundle/programs/server/npm/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:36:498)
2019-10-03 21:15:56-04:00 at Za (/app/bundle/programs/server/npm/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:39:16)
2019-10-03 21:15:56-04:00 at a.b.render (/app/bundle/programs/server/npm/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:476)
2019-10-03 21:15:56-04:00 at a.b.read (/app/bundle/programs/server/npm/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:18)
2019-10-03 21:15:56-04:00 at renderToString (/app/bundle/programs/server/npm/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:54:364)
2019-10-03 21:15:56-04:00 at Promise.asyncApply (server/ssr.js:40:12)
2019-10-03 21:15:56-04:00 at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:u5408:

That error means a hook was called outside of a component - and it was not (it appears to have been called inside the Loadable component).

Hmmm, Loadable uses an internal psuedo-global store for determining if Loadables have been loaded. I wonder if the Fibers could be messing with that.

1 Like

@captainn how is it going?

I’m a bit stuck, and simply out of time to work on this for the moment. I think there is some kind of bad interaction happening between Fibers and React hooks, but I’m not 100% certain. The only real feedback I have going so far is the stack trace I posted earlier. It shows something happening in my Loadable component, which uses hooks and the root of the stack trace comes from the fiber pool. … I just thought of something. Maybe my loadable is leaking. Investigating…

1 Like

Could this be creating a memory leak?

const LoadableContext = createContext(false)
export const LoadableCaptureProvider = ({ handle, children }) => {
  if (!handle.loadables) {
    handle.loadables = [] // this in particular
    handle.toEJSON = () => ( // or maybe these closures
      EJSON.stringify(handle.loadables)
    )
    handle.toScriptTag = () => (
      `<script type="text/ejson" id="__preloadables__">${EJSON.stringify(handle.loadables)}</script>`
    )
  }
  return createElement(LoadableContext.Provider, { value: handle }, children)
}

That provider sets a number of values meant to help with capturing some data when running in SSR. It runs each time the server-render package’s onPageLoad is envoked, to create a new set of state for the current render. Is it possible that something about this pattern is preventing the array or other enclosed values from being collected?

hmmm… some thoughts:

  • the cpu spikes when memory is short could be garbage collecting and / or swapping, so probably not a problem on its own
  • the context issue mentioned above with React.createContext is surely not the problem (and fixed), it just lead to wrong context values
  • Maybe you have some client-code that stores stuff in global variables which is now run in the server and every request might fill that global variable. This can happen for code that is not really optimized for server run. Or has to be treated differently on server. E.g. take some cache like apollo-cache. On the client, you can initialize apolloClient and its cache globally, as some singleton. But you have to be careful when doing SSR: you should treat every request as a separate entity, so you should initialize the apolloClient and its cache in this request and its function. Once this request has sent data to the client, everything initialized should get garbage collected. So i would check if you initialize someting globally and not within the server-render sink. Check if you have anything that might cache stuff (e.g. a i18n-library). Meteor-collection can be global on server, because they are isomophically designed, but maybe you use some library around it, that is not aware of SSR.
  • maybe it has to do with the loadable code above, but it does not seam to add so much data… the code is a bit weird. how does LoadableCaptureProvider receive a handle?
  • try to reproduce it locally, maybe copy the whole database and check memory consumption after every render? there must be something that you can reproduce

Not sure if this is your problem or if this will be helpful, but https://blog.meteor.com/an-interesting-kind-of-javascript-memory-leak-8b47d2e7f156 is an article I refer to every once in awhile whenever I see a weird memory leak.

It’s old and perhaps no longer relevant… it’s just something that stuck with me because it looked like such an easy thing to do.

I’m fairly sure I’m not creating any global (or package level) variables. This seems to be leaking out of the Mongo queries. I’m using my collections package, which uses a separate code base for the server and the client. On the server I’m using Mongo queries directly, which means behind the scenes it’s using Fibers. Since the fiber pool file is at the bottom of the stack trace (the error I get right before “out of memory” in server logs) I have been thinking it has something to do with some interplay between Fibers and React SSR.

Could parts of the React render be getting garbage collected before the hook is run? It seems weird that React tells me I’m calling a hook outside of the component tree, when the component I’m calling it from is right there in the stack.

Reproducing it has been challenging - I actually can’t even reproduce it on the same service (Galaxy) when hooked up to the same production Mongo database, with a different URL. That’s very frustrating.

If I do let it run locally and grab a heap dump, I see a lot of Mongo driver allocations, but nothing super obvious.

The handle is created in the SSR code, and then handed to the Provider, which sets up the handy tools, like toScriptTag. I assumed that would make it GC safe, but I’m not so sure.

import { WebApp } from 'meteor/webapp'
import React from 'react'
import s2s from 'string-to-stream'
import sq from 'streamqueue'
import { StaticRouter } from 'react-router'
import { renderToString } from 'react-dom/server'
import { FastRender } from 'meteor/staringatlights:fast-render'
import { LoadableContext, preloadAllLoadables } from 'meteor/npdev:react-loadable'
import { HelmetProvider } from 'react-helmet-async'
import { ServerStyleSheets, ThemeProvider } from '@material-ui/styles'
import App from '/imports/App'
import theme from '/imports/ui/common/theme'
import { DataCaptureProvider } from 'meteor/npdev:collections'
import { EJSON } from 'meteor/ejson'

preloadAllLoadables().then(() => FastRender.onPageLoad(async sink => {
  const context = {}
  const helmetContext = {}
  const dataHandle = {}
  const loadableHandle = { loadables: [] }

  const sheets = new ServerStyleSheets()

  const app = <ThemeProvider theme={theme}>
    <HelmetProvider context={helmetContext}>
      <LoadableContext.Provider value={loadableHandle}>
        <DataCaptureProvider handle={dataHandle}>
          <StaticRouter location={sink.request.url} context={context}>
            <App />
          </StaticRouter>
        </DataCaptureProvider>
      </LoadableContext.Provider>
    </HelmetProvider>
  </ThemeProvider>

  let html
  try {
    html = renderToString(sheets.collect(app))
  } catch (e) {
    console.error(e)
    WebApp.addHtmlAttributeHook(() => ({
      lang: 'en'
    }))
    return
  }

  const { helmet } = helmetContext
  const meta = helmet.meta.toString()
  meta && sink.appendToHead(meta + '\n')
  const title = helmet.title.toString()
  title && sink.appendToHead(title + '\n')
  const link = helmet.link.toString()
  link && sink.appendToHead(link + '\n')

  WebApp.addHtmlAttributeHook(() => (
    Object.assign({
      lang: 'en',
      'xmlns:og': 'http://ogp.me/ns#'
    }, helmet.htmlAttributes.toComponent())
  ))
  // :TODO: Figure out how to do helmet.bodyAttributes...

  const css = sheets.toString()
  sink.appendToHead(`<style id="jss-server-side">\n${css}\n</style>`)

  // :HACK: The meteor css bundle should come after the JSS output, so we just move it manually
  sink.appendToHead('\n<script id="css-fixer">elm=document.getElementsByClassName("__meteor-css__")[0];elm.parentNode.appendChild(elm);elm=document.getElementById("css-fixer");elm.parentNode.removeChild(elm);delete elm</script>')

  const queuedStreams = sq(
    () => s2s(html),
    () => s2s(`<script type="text/ejson" id="__preloadables__">${EJSON.stringify(loadableHandle.loadables)}</script>`),
    () => s2s(dataHandle.toScriptTag())
  )
  sink.renderIntoElementById('root', queuedStreams)
}))

Actually, this is half way through a refactoring I’m doing to test whether my libraries are leaking the memory.

I think that was it! Here is code that doesn’t seem to leak (so far):

import { WebApp } from 'meteor/webapp'
import React from 'react'
import s2s from 'string-to-stream'
import sq from 'streamqueue'
import { StaticRouter } from 'react-router'
import { renderToString } from 'react-dom/server'
import { FastRender } from 'meteor/staringatlights:fast-render'
import { LoadableCaptureProvider, preloadAllLoadables } from 'meteor/npdev:react-loadable'
import { HelmetProvider } from 'react-helmet-async'
import { ServerStyleSheets, ThemeProvider } from '@material-ui/styles'
// import { green, red } from '@material-ui/core/colors'
import App from '/imports/App'
import theme from '/imports/ui/common/theme'
import { DataCaptureProvider } from 'meteor/npdev:collections'

h = React.createElement // eslint-disable-line

preloadAllLoadables().then(() => FastRender.onPageLoad(async sink => {
  const context = {}
  const helmetContext = {}
  let dataHandle = {}
  let loadableHandle = {}

  const sheets = new ServerStyleSheets()

  const app = <ThemeProvider theme={theme}>
    <HelmetProvider context={helmetContext}>
      <LoadableCaptureProvider handle={loadableHandle}>
        <DataCaptureProvider handle={dataHandle}>
          <StaticRouter location={sink.request.url} context={context}>
            <App />
          </StaticRouter>
        </DataCaptureProvider>
      </LoadableCaptureProvider>
    </HelmetProvider>
  </ThemeProvider>

  let html
  try {
    html = renderToString(sheets.collect(app))
  } catch (e) {
    console.error(e)
    WebApp.addHtmlAttributeHook(() => ({
      lang: 'en'
    }))
    return
  }

  let { helmet } = helmetContext
  const meta = helmet.meta.toString()
  meta && sink.appendToHead(meta + '\n')
  const title = helmet.title.toString()
  title && sink.appendToHead(title + '\n')
  const link = helmet.link.toString()
  link && sink.appendToHead(link + '\n')

  WebApp.addHtmlAttributeHook(() => {
    const attrs = Object.assign({
      lang: 'en',
      'xmlns:og': 'http://ogp.me/ns#'
    }, helmet?.htmlAttributes?.toComponent() || {})
    helmet = null
    return attrs
  })
  // :TODO: Figure out how to do helmet.bodyAttributes...

  const css = sheets.toString()
  sink.appendToHead(`<style id="jss-server-side">\n${css}\n</style>`)

  // :HACK: The meteor css bundle should come after the JSS output, so just move it manually
  sink.appendToHead('\n<script id="css-fixer">elm=document.getElementsByClassName("__meteor-css__")[0];elm.parentNode.appendChild(elm);elm=document.getElementById("css-fixer");elm.parentNode.removeChild(elm);delete elm</script>')

  const queuedStreams = sq(
    () => s2s(html),
    () => {
      html = null
      return s2s(loadableHandle.toScriptTag())
    },
    () => {
      loadableHandle = null
      return s2s(dataHandle.toScriptTag())
    },
    () => {
      dataHandle = null
      return s2s('')
    }
  )
  sink.renderIntoElementById('root', queuedStreams)
}))

I am getting some sync errors on hydration, but it’s no longer leaking! I’ll investigate sync errors later.

1 Like

Update, this didn’t solve the memory leak completely, but it did slow it way down! I’ll keep looking through the code for similar places to improve, and report back if I find anything useful to share.

3 Likes

for reference, here is a (somewhat messy) ssr code snipped we use (unaltered and undocummented). It also uses a page cache in mongodb:

import { FlowRouter } from 'meteor/ostrio:flow-router-extra';
import { FastRender } from 'meteor/staringatlights:fast-render';
import { InjectData } from 'meteor/staringatlights:inject-data';
import { Random } from 'meteor/random';
import { Meteor } from 'meteor/meteor';
import moment from 'moment';
import { renderToString } from 'react-dom/server';
import React from 'react';
import Radium from 'radium';
import { Helmet } from 'react-helmet';
import Loadable from 'react-loadable';
import { PageNotFoundError } from '/imports/api/errors';
import { PageCache } from '/imports/api/collections';

import app from '/imports/modules/app';

InjectData.disableInjection = true;
const RadiumWrapper = Radium(
  class extends React.Component {
    /* eslint react/destructuring-assignment: 0 */
    render() {
      return this.props.children;
    }
  }
);

app.init();

const send404 = sink => {
  sink.setStatusCode(404);
};

const renderTheRoute = ({ sink, Component, props, matchinRoute }) => {
  // we need to keep the random seed accress cached pages
  // as the random seed has only visual implications, that's not a big problem
  // result is that every page with the same random seed looks the same
  const randomSeed = Random.id();
  const modules = [];
  const modulesResolved = [];
  const { url, query: queryParams } = sink.request;
  const { pathname: path } = url;
  const { route, params } = matchinRoute;

  FlowRouter.setCurrent({
    path,
    params,
    route,
    queryParams,
  });

  const appString = renderToString(
    <Loadable.Capture
      report={moduleName => {
        modules.push(moduleName);
      }}
      reportResolved={resolvedModuleName => {
        modulesResolved.push(resolvedModuleName);
      }}
    >
      <RadiumWrapper radiumConfig={{ userAgent: 'all' }}>
        <Component {...props} randomSeed={randomSeed} />
      </RadiumWrapper>
    </Loadable.Capture>
  );

  const helmet = Helmet.renderStatic();
  const headParts = [
    helmet.title.toString(),
    helmet.meta.toString(),
    helmet.link.toString(),
  ];

  // see https://github.com/abecks/meteor-fast-render/issues/23
  FastRender._mergeFrData(sink.request, FastRender.frContext.get().getData());
  const fastRenderData = sink.request.headers._injectPayload;
  const bodyParts = [
    `<script> var __randomSeed = "${randomSeed}";</script>`,
    `<script> var __preloadables__ = ${JSON.stringify(
      modulesResolved
    )};</script>`,
    `<script type="text/inject-data">${InjectData.encode(
      fastRenderData
    )}</script>`,
  ];

  return { bodyParts, headParts, appString };
};

const getCacheKey = sink => `${sink.request.url.path}`;

const sendMatchingRoute = ({ matchinRoute, sink, path, queryParams }) => {
  const requestId = Random.id();
  const { route, params } = matchinRoute;
  if (route.options?.triggersEnter?.length > 0) {
    // this is not totally correct as we should respect triggers stop and redirect
    // we use it only for i18n route, so for us its fine i guess
    route.options.triggersEnter.forEach(trigger =>
      trigger({ ...route, params }, () => null, () => null)
    );
  }

  route.render = function(Component, props) {
    // currently disabled
    const useCache = !process.env.CACHE_DISABLED && !Meteor.userId();
    if (useCache) {
      const cacheKey = getCacheKey(sink);

      if (
        !PageCache.findOne({
          _id: cacheKey,
          updatedAt: {
            $gte: moment()
              .subtract(5, 'minutes')
              .toDate(),
          },
        })
      ) {
        const doc = {
          _id: cacheKey,
          updatedAt: new Date(),
          ...renderTheRoute({
            Component,
            props,
            sink,
            matchinRoute,
          }),
        };
        console.log('cache update');
        PageCache.upsert(cacheKey, { $set: doc });
      } else {
        console.log('cache hit');
      }
      const { headParts, bodyParts, appString } = PageCache.findOne(cacheKey);

      headParts.forEach(part => sink.appendToHead(part));
      bodyParts.forEach(part => sink.appendToBody(part));
      sink.renderIntoElementById('react-root', appString);
    } else {
      console.log('no cache will be used (logged in)');
      const { headParts, bodyParts, appString } = renderTheRoute({
        Component,
        props,
        sink,
        matchinRoute,
      });
      headParts.forEach(part => sink.appendToHead(part));
      bodyParts.forEach(part => sink.appendToBody(part));
      sink.renderIntoElementById('react-root', appString);
    }
  };

  const data = route.data ? route.data(params, queryParams) : null;

  console.log('================');
  console.log(`render-${requestId}, ${path}`);
  console.time(`rendertime-${requestId}--${path}`);

  try {
    route.action(params, queryParams, data, { sink });
  } catch (e) {
    if (e instanceof PageNotFoundError) {
      send404(sink);
    } else {
      console.error('server render failed', e);
    }
  }
  console.timeEnd(`rendertime-${requestId}--${path}`);
  console.log('================');
};

Loadable.preloadAll().then(() => {
  FastRender.onPageLoad(async sink => {
    const { url, query: queryParams } = sink.request;
    const { pathname: path } = url;

    if (path && path.startsWith('/__')) {
      // skip internal endpoints like __cordova
      // see https://github.com/meteor/meteor/issues/10557
      return;
    }
    const matchinRoute = FlowRouter.matchPath(path);
    if (matchinRoute) {
      sendMatchingRoute({
        matchinRoute,
        sink,
        path,
        queryParams,
      });
    } else {
      send404(sink);
    }
  });
});

1 Like