Store settings in database or JSON

help
1

Hello! Are there any guidelines on when settings should be stored in the database vthe s settings.json file? Specifically, I have three applications which are talking to the same database. Some settings, such as Quickbooks account refids, need to be stored somewhere, and they are different between production and development instances. Today we put them in .json, but if i need to change them, then I need to edit three separate files and redeploy three separate applications. I suppose this is a good reason to move such settings into a database. Perhaps I put them in a mongodb “appSettings” collection under different ids and use the development.json or production.json file to designate which settings doc to pull from the db?

{ 
    "_id" : "development", 
    "perItem" : 100, 
    "perShop" : 400, 
  "quickbooks": {
    "isSandbox": true,
    "serviceCodes": {
      "INCOME_BSP_CONSOLIDATIONS_FEE": 22,
      "INCOME_BSP_MAILOUT_FEE": 23,
      "INCOME_BSP_RECEIVING_FEE": 24
    }
  }
}

{ 
    "_id" : "production", 
    "perItem" : 100, 
    "perShop" : 400, 
  "quickbooks": {
    "isSandbox": false,
    "serviceCodes": {
      "INCOME_BSP_CONSOLIDATIONS_FEE": 53,
      "INCOME_BSP_MAILOUT_FEE": 54,
      "INCOME_BSP_RECEIVING_FEE": 51
    }
  }
}
1 Like
2

This is interesting… I usually keep them in a settings file but it seems putting them in the DB will be more scalable.

I’m guessing by three applications, you have three different clients all using the same app/product? So right now you just adjust the settings for each client, and deploy a separate server for each of them?

1 Like
3

This is something I’ve encountered on two projects now, where we initially started with multiple settings files (for different clients/environments) and ended up moving to a settings collection in the database. My reasons:

  1. Re-deplolying to update a setting is not a great situation.

  2. Trying to debug/test settings (see #1).

  3. If you’re caught away from your dev environment, you can still log into mLab (or wherever) to update a setting.

  4. (IMO) Maintaining more than 2-3 settings files is tedious and error-prone.

I think the settings.json file is a great utility for probably 90% of apps/use cases, but it has its limitations.

1 Like
4

how did you approach things? Did you just move the settings json into a collection? One of my devs suggest I flatten the values to make them a simple dictionary (name/value pairs). But we use some of the values in ENUMS so this is going to require quite a bit of refactoring

BEFORE

{
  "quickbooks": {
    "isSandbox": true,
    "serviceCodes": {
      "QBO_INCOME_BSP_CONSOLIDATIONS_FEE": 22,
      "QBO_INCOME_BSP_MAILOUT_FEE": 23,
      "QBO_INCOME_BSP_RECEIVING_FEE": 24,
      "QBO_INCOME_BSP_DISCOUNT_GENERIC": 27,
      "QBO_INCOME_BSP_SPECIAL_HANDLING": 41
    },
    "bankAccounts": {
      "QBO_BANKACCOUNT_PAYPAL": "148",
      "QBO_BANKACCOUNT_STRIPE": "149"
    },
    "paymentMethods": {
      "QGO_PAYMENTMETHOD_STRIPE": "10",
      "QGO_PAYMENTMETHOD_PAYPAL": "11"
    }
  }
}

AFTER

  "development" :[{
  "QBO_ISSANDBOX" : true,
  "QBO_ITEM_CONSOLIDATIONS_FEE": 22,
  "QBO_ITEM_MAILOUT_FEE": 23,
  "QBO_ITEM_RECEIVING_FEE": 24,
  "QBO_ITEM_DISCOUNT_GENERIC": 27,
  "QBO_ITEM_SPECIAL_HANDLING": 41,
  "QBO_BANKACCOUNT_PAYPAL": "148",
  "QBO_BANKACCOUNT_STRIPE": "149",
  "QBO_PAYMENTMETHOD_STRIPE": "10",
  "QBO_PAYMENTMETHOD_PAYPAL": "11",}]

But then it makes it harder to do things like this

const {INCOME_BSP_SHIPPING_RATE, INCOME_BSP_MAILOUT_FEE, INCOME_BSP_RECEIVING_FEE} = Meteor.settings.quickbooks.serviceCodes;
5

We use environment variables for this which seems to be the right way. We have developer defaults, so when the environment variables are not available we use those automatically. That way we do not have to store logins etc. in

We also have a config file per deployment, for fixed configurations which do not change based on system administrators. We automatically deploy with a .sh script, which just calls galaxy deploy with the right config file.

So we make a difference in settings. For example:

Environment variables:

  • DB login
  • Mailgun login
  • Etc.

Config file:

  • _id: Since it’s fixed.
  • Quickbooks.isSandbox, you never want to change it to production on your development.
  • Your servicecodes I am not sure of, depends on what it is and who can change it. If it’s configurable in quickbooks I suggest environment variables.

Those service codes might be a candidate for database storage but only when they are configurable from for example an admin. We never store settings in the database which should not be changed by the app.

The focus on environment variables comes partly from: https://12factor.net/config but they have an even more strict focus on using environment variables. We like the fact that we have a config file for setting the behavior of functional changes so they are powered by the developer, not the sys admin.

6

where are environmental variables stored? in the app right? that doesn’t solve the problem. we have multiple apps connecting to the same database. Some settings which are shared, could be in settings files or environmental vars but they you have to change them in multiple places and redeploy multiple apps. The whole point was that putting them in the db would prevent that. Or am I missing something? or maybe you missed the original context?

7

I get your context but I still would consider the approach as described.

To be clear, environment variables are configured in your hosting environment, like Galaxy. They are not part of your software but part of the environment.

If you need to deploy the same config to multiple places you can create some kind of update script for that if you need to do it centrally. In bigger environments software like Puppet is being used for that.

In general it’s good to restart after the environment changes, that’s also what Galaxy does. Since you need for example a new database connection. Also they should not change often.

Your quickbooks example depends on the implementation and features. It could make sense to store those serviceCodes in the database. And sometimes it does not, it depends.

If you put them in the database make a simple admin screen for them. That way you don’t need manual access to the database which will become an issue in future otherwise. And your code should validate that they are set and give an appropriate warning when not available.

8

Storing settings in a database would be very convenient, just make sure you do not store any credentials though… That would even provide reactivity which can be as good as it can be bad if an application needs to take action when something changes, the dev has to be careful (invalidating caches and the like maybe?)

For my part, I have 4 applications running very much hand in hand. I have 1 app that just shows the login, to make it load fast, once the user is logged in he is transferred to the “members section” app. From this app, they can reach a third one which is some heavy tool to work on the user’s stuff. The fourth app is completely hidden from the users, it is basically just handling CPU heavy jobs.

I needed a way to monitor/scale the 4th app and keep all apps aware of some global settings and using a database didn’t seem to be fine at first sight (but I might have been wrong). What I did instead is use my login app, as an arbiter too. Since it is lightweight and highly available, this would make it perfect to store settings and provide other apps with them.

So I store my settings in one of four apps, the other 3 apps use DDP to request settings. Once a request has been served, the arbiter remembers who needs what and sends notifications (again, over DDP) of any changes so they can react to it as required.

I do not think this is the best idea. This was made fast as we needed a way to communicate between the apps easily but keep it centralised. So we used the same feature to share settings as we’re like you, we didn’t want to redeploy everything. So far it’s running ok (note I’m only saying ok) and we haven’t launched the app yet… I wanted to post this so you get the info and maybe other ideas from it, but thinking back we’ll certainly switch to using a database too after we launch.

1 Like
9

thanks for sharing @Salketer!

@lucfranken yeah, to be clear, I understand the difference between different kinds of settings, so yes, confidential stuff like API keys will remain in settings (or could be migrated to env vars). It’s just some things which might change, like if we add a new service, we’ll need to add new quickbooks items to track it, so we’ll need to add a refId and deploy new apps. So that stuff should be in the db. Unfortunately I have a bit of refactoring to do to make that happen. Maybe we’ll make a global settings NPM app and share it for others to use.

10

@maxhodges , I’ve exchanged a word about this with the guy next to me in
the open space and he sighted. “How will he get his apps notified when he
turns the ‘db_maintenance’ token on and brings down their databases?” I
found it pretty self explanatory… You need something 100% outside of what
your app uses to work.

I’d vote for using Redis, but any distributed key-value store could be
enough. Using another MongoDB could also work I guess.

11

As I mentioned, we won’t put all the values in the database. API keys, db connection strings, will be in settings.json (or environmental variables might work). We’ll use the db for shared settings, but not things like my production stripe key!

Tell him we don’t currently have anyway to put the application in maintenance mode. We’ve never had to take it down in 2.5 years because of the way Galaxy and Meteor orchestrate hot code pushes! it may even be possible to migrate to a new database host without downtime–you could have your new database use the former database’s replication set until you get things migrate. It’s a whole new era!

12

btw for your “CPU heavy jobs” app, perhaps you could use something like https://webtask.io/

13

Sorry for the delayed response, as I was traveling.

We found that having two settings collections is a good solution, depending on your use case. One collection was basically a features collection, where we could enable/disable features on a per-client basis, and have our ui/client experience depend on flags and docs within that collection. Something like:

{
  featureOneEnabled: true,
  featureTwoEnabled: false
}

And then basically just do a subscription in the main layout template to the main doc in that collection.

We also have a second collection where we store sensitive client information. There are no pub/subs for this collection, and we are only able to access those docs via server-side code. Of course, some of those methods can be called from the client, but that requires auth.

Hope that helps!

1 Like
14

I am not an expert and did not read all posts. One option I am evaluating for my project, is to call a server side method that would import the desired file and return the content to the user. It can filter according to user and context, serve as need to know. But each app needs to have access to the same file.

(My use case is about metadata for customizing generic vue.js components, such as one form for several contexts, down to details about each control on the form, and according to user permissions, this allows to reduce code over the wire and highly reusable components)

Another concern is that if the file changes, how could I communicate in a controled rate to all ‘users’ that they need to query again, other than asking if need update every hour, or manual requery of data. (controled rate because I do not want all users to update at the same time, such as in a live mongoDB query)

Finally, there is the question of offline availability on client side.

15

why not use the database instead of a file? Then you’d get automatic updates via meteor’s pub/sub framework