Evaluating Meteor as an API server

help
1

Continuing the discussion from Running multiple instances of Meteor with separate databases while being able to push updates to each:

Hey @aadams I already feel like I’ve hijacked that original thread so hopefully splitting this to another topic will help follow the two separate topics.

First, thank you very much for your kind words, I do sincerey hope this is helpful for you.

Ok so let’s first clear up the webapp/connect/express topic. WebApp is a Meteor core package that deals with responding to http requests, this is what sends down the initial html/css/js bundles and sets up the initial connection for ddp to work.

Nodejs already has a http api which has a createServer method that sets up an http server for you. This is, in essence, all you need for a rest api because it enables you to listen for incoming http protocol requests and respond to them.

But the api is too raw for most people’s taste/needs. Most projects need a way to do a few chained operations on the incoming request, like parse the url, parse the body, set some headers, log a few things etc. Since most of these small operations are really common and shared across the whole set of endpoints, the term middleware was coined. So someone built a library that made it easy to define a shared api and connect these independent middleware functions, and called it connect.

But there were another set of more specialized apps that needed more on top of this, web apps! They typically needed a built in router, a view engine, ability to serve static files and an _express_ive way to wire these all up so that same someone created express, building on top of connect.

Aside: There’s a whole complicated history around connect and express and currently, express is no longer dependent on connect, although the middleware api remains 100% compatible making most code from one reusable in the other, all the while the original developer of both has moved on to Koa and then stopped using node alltogether in favor of Go :wink:

Now I know that express is a lot more popular (github stars, npm downloads etc) but connect is also as well maintained and my personal preference lies with libraries that have smaller api surfaces whenever they’re sufficient. And I find connect to be 100% sufficient for most rest api projects. I use a router middleware to make working with routes easy and I’m covered.

You can of course use express, but to me it feels like buying a swiss army knife for its screw driver while there already is a screw driver lying around, which is WebApp.connectHandlers() which is connect in half-disguise!..

So, out of box, Meteor indeed “does offer a restful api” because it has no less to offer than what a barebones express app offers, a single route that matches / and responds to http requests!

Now, moving on to the ddp/rest/method/graphql discussion.

So the question is, what api(s) should one offer from a server side (possibly meteor) application?

To answer that question, you should look at the capabilities of the client apps that would connect to your server app.

Obviously, Graphql (possibly via Apollo) is an increasingly popular choice, while REST is kind of an hard to miss target for the forseable future and DDP (pub/sub and methods) is a very productive choice if you can get the clients to speak DDP.

Aside: Those clients would be your meteor app’s frontend, other meteor apps’ server and frontends, any javascript client using a third party ddp library, and again using a third party library, ios and android native apps, react native apps

If you’re already invested in a meteor application and simply would like to increase client coverage, you can use atmosphere packages like nimble:restivus or simple:rest which both have the ability to expose your methods/publications as rest endpoints without much effort.

This could very well be a stop gap solution until you refactor your application to separate your “application logic” from your “api protocols/endpoints” such that you can reuse the application logic to create any one or more of the alternative api facades and be safe even when something new becomes the new groundbraking api protocol next year! :slight_smile:

If you are starting from scratch, though, that’s another story which requires you to do some research and surveying to figure out your application ecosystem.

In any case, though, I find that Meteor caters very well to all of these options if you can bring yourself to think of it as a build tool that gives you all the tools, libraries, ecosystem access and language features you need to build a modern javascript server and/or web and/or mobile and/or desktop application with a single meteor create command.

Final note: As with all technology stack, languages etc, there will probably be one point where your scale or specific requirements will call for including other small or large pieces of tools to solve some specific aspect of your application domain. There’s no one size fits all but this one-size - meteor - is quite dependably versatile.

9 Likes
2

You are a legend! Thank you for your feedback here. It does really help and put things into context for anyone thinking of attempting doing something like this – which I agree Meteor is well suited to.

WebApp is a Meteor core package that deals with responding to http requests, this is what sends down the initial html/css/js bundles and sets up the initial connection for ddp to work.

I always wondered where in the core this all happened, interesting.

Nodejs already has a http api which has a createServer method that sets up an http server for you. This is, in essence, all you need for a rest api because it enables you to listen for incoming http protocol requests and respond to them.

I’ve seen examples where they set up a web service without Express – the examples I saw were all every basic/rudimentary. For anything more advanced in the way of Restful API I have always seen Expressed used. It sounds like outside of File system work, this webserver work is one of the basic core functions of Nodejs – not taking into account Node’s event and queueing system.

But the api is too raw for most people’s taste/needs. Most projects need a way to do a few chained operations on the incoming request, like parse the url, parse the body, set some headers, log a few things etc. Since most of these small operations are really common and shared across the whole set of endpoints, the term middleware was coined. So someone built a library that made it easy to define a shared api and connect these independent middleware functions, and called it connect.

I think I’ve seen a few examples years back where connect was used. I really didn’t understand what connect was doing back then, now I have a better idea. I wonder if Meteor and/or Apollo internally use connect and express to handle middleware and RESTful API calls.

But there were another set of more specialized apps that needed more on top of this, web apps! They typically needed a built in router, a view engine, ability to serve static files and an expressive way to wire these all up so that same someone created express, building on top of connect.

And I think this is what I’d like to replicate, with Meteor (server side only). I’d like to build a great RESTful API that can MOSTLY be called by Meteor client applications either locally from the same server and eventually from remote servers. I would also like to be able to call this service from a iOS device – but that’s far down the road.

This Meteor server side application with a RESTful API (I’ll call it a Meteor Worker Service or MWS for short) – will handle long(er) running applications that many clients will make requests to. After the MWS is done it will place resulting file(s) on a cloud file service for the clients to pick up. This MWS will need to have access to all the different Meteor client MongoDB databases to pull in the relevant data. These MongoDB database are all cloud services with replica sets that are accessed with a URL (the typical MongoDB cloud service). I think of it like a hub and spoke architecture – with the MWS in the middle and the MonogDBs and Meteor client applications surrounding it.

These Meteor clients that call out to the MWS will also each of course and as usual have their own Meteor server side (just Nodejs) that they make Meteor method calls to and I’m still using the Sub/Pub pattern here as well. Each with their own MongoDBs accessed by the URLs – the same DBs that the MWS should access when the API calls triggered and it’s so instructed.

Now I know that express is a lot more popular (github stars, npm downloads etc) but connect is also as well maintained and my personal preference lies with libraries that have smaller api surfaces whenever they’re sufficient. And I find connect to be 100% sufficient for most rest api projects. I use a router middleware to make working with routes easy and I’m covered.

It sounds like we have the same tendencies when it comes to using simplified libs over complex or bloated ones – for various reasons. One of which is that if the lib is too complex, and becomes abandoned, it will be difficult for me to recover. And of course if the lib is more simple, the opposite is true.

You can of course use express, but to me it feels like buying a swiss army knife for its screw driver while there already is a screw driver lying around, which is WebApp.connectHandlers() which is connect in half-disguise!..

You mention here that Express has been abandoned – years ago I remember reading about when the community was up in arms about this and speculating about what came next. Koa seemed to never take over like they thought it would – instead Express has lived on and is still widely used. That does provide some comfort.

Having said that, I wouldn’t mind using “raw” connect or restivus – if there were a healthy number of examples of doing in the Meteor context (I know restivus is only Meteor). Using the connect lib to build a Meteor server RESTful API I think is a edge-case and there will be little to no “support” (in the form of articles/videos/forum posts/etc) if I go this route. And since connect is more “raw” there will be more “knowledge” of how things work “under the hood” so to speak that will be necessary. Come pare that to restivus (maybe) or Express.

For someone at your level, someone how understands the internals, using “raw” connect isn’t a problem – for a business owner and tech enthusiast like me that doesn’t have a ton of time to research how the “motor” and related system in the car works – just knows how to drive (sometimes fast) – it can become a problem drifting too far off in edge-cases. My heart and mind is drawn by curiosity and excitement to try to understand what’s going on internally, to try to use the more “raw” tools that do less and require more of me – but my business side turns me away from that.

Obviously, Graphql (possibly via Apollo) is an increasingly popular choice, while REST is kind of an hard to miss target for the foreseeable future and DDP (pub/sub and methods) is a very productive choice if you can get the clients to speak DDP.

I wouldn’t mind using Graphyql (or Apollo) to access the MongoDB collections from MWS. Also, right now, all the clients use standard (old school) DDP connections to their respective servers; using Meteor methods and pub/sub when appropriate. This works, if I need to be reactive I use pub/sub and for everything else I try to use Meteor methods. It’s only when I need these long running tasks that the MWS will come into play.

If you’re already invested in a meteor application and simply would like to increase client coverage, you can use atmosphere packages like nimble:restivus or simple:rest which both have the ability to expose your methods/publications as rest endpoints without much effort.

This might be a better option than using Express. Yet both restivus and simplerest seems abandoned. And when these smaller projects are abandoned, unlike Express, I’ve seen a pattern where there’s just no reviving them – if there’s a bug or problem, it could become a problem.

This could very well be a stop gap solution until you refactor your application to separate your “application logic” from your “api protocols/endpoints” such that you can reuse the application logic to create any one or more of the alternative api facades and be safe even when something new becomes the new groundbraking api protocol next year!

I already have the MWS functionality in every single application I’ve deployed to my clients. But it seems like an application living within the application that has it’s own unique local file system and cloud file system requirements, cpu and ram requirements (the box has to be larger for each client to handle the load). It’s a heavy load for each application, in terms of infrastructure, files, external executable dependencies, deployments, and is prone to issues if I have a “lighter” client. It would be better for me to centralize this portion into the MWS where I can maintain it in one place, and just “service” all the existing Meteor clients. It will take work factoring out the code from the clients – but once that’s done, I have a lot more flexibility.

In any case, though, I find that Meteor caters very well to all of these options if you can bring yourself to think of it as a build tool that gives you all the tools, libraries, ecosystem access and language features you need to build a modern javascript server and/or web and/or mobile and/or desktop application with a single meteor create command.

I agree! I’m going to trying to refab Meteor into a server side worker service with RESTful API! :smiley:

Thank you for all your help so far!

1 Like
3

You’re right on with this. Great write up on your experience here.

2 Likes
4

Based on your reply, I’d advise you to go with simple:rest or nimble:restivus if that’s also too barebones for you, because while simple:rest provides you a very easy rest endpoints on top of your methods, nimble:restivus is more feature rich.

In either case, I would not worry about them not being maintained because honestly, they don’t require much maintenance, they are both quite mature and stable.

This would serve you as a nice stop gap solution to automagically offer rest apis to your clients and that would give you more time to investigate your other options.

In the meantime, ı’d also advise keeping your workers as meteor apps becuase that would allow them to share the same api surface that your main app(s) utilize. Furthermore, you can connect multiple meteor server apps together through a common database or ddp connections between each other, allowing you to develop ini reactive style.

1 Like
5

Thank you for the advise! I wil keep my workers as Meteor apps and l try nimble:restivus over Express based on your recommendation. :smiley:

1 Like
6

Make sure you spare a little time to evaluate simple:rest beforehand because it is much smaller and might be surprizingly sufficient for you.

1 Like
7

I came away disappointed with all the open issues and PRs laying in wait on both the restivus and simplerest libs.

.
Some thought instead of using restivus or simplerest, I should just use Express or Restify.

711×512 72 KB

The above converstation came about becasue I was considering rebuilding restivus in order to remove the coffee script, add it to NPM, and better understand the code. I really don’t want to take on an abandoned lib, so if there’s a possibility I can maintain it myself, then I would use it. Some thought this wasn’t the smartest idea.

.
Here’s the project just in case: GitHub - aadamsx/fine-rest: fine-rest: make your app accessible over HTTP and DDP

.
I put in a Meteor feature request for a REST in core here just in case a miracle is possible there:

.
Also just trying to better understand here, I take it your advice to @firrae is to crack open the source code on some or all of the simplerest packages and kludge together a custom solution from that?

.
@serkandurusoy, did you just use connect to build your Meteor API?

2 Likes
8

We’re running an API server using JsonRoutes and FHIR resources. FHIR makes it easy for us, because all the endpoints and schemas are already defined… basically a giant healthcare API we just implement to. OAuth was a hassle, and I’m surprised more people in the Meteor community aren’t running OAuth servers. We were only able to get it working because JsonRoutes includes the connect package. Collection hooks are our friends, and enable nicely decoupled hooks to inbound/outbound API calls. $0.02

2 Likes
9

Personally speaking, we have decided to focus on GraphQL as our main endpoint to our mobile app but auth was still an issue for us as our mobile app is written in C#, not JS. Overall it wasn’t hard to take the concepts in simple:rest and apply the auth process to GraphQL and as such I don’t forsee any issues making it work in Express. Our commercial REST API for customers will likely end up being Express based as there are a few things I want to do that I have prototyped out in Express already. When we go to move this I will likely look at publishing an Express middleware, if someone doesn’t beat me to it, that imitates @sashko’s middleware in simple:rest onto NPM to make it easier for others.

Our alternative was to just get a Connect router and plug it onto the Meteor WebApp on a sub path as @serkandurusoy implied in the OP but there is stuff in the pipe for us that makes Express look more appealing as we want to add stuff in the future.

2 Likes
10

Yes, indeed I did. But the reason I suggested you begin with simple:rest or nimble:restivus was that you said you were not experienced enough to take on writing apis on your own, so one of those two might be a stop gap solution.

As for the comparison between connect/express, as I said, that depens on whether or not you need the extra individual apis/features that express offers on top of the connect api, which I never needed.

As for the PR you’ve created, my opinion is that meteor already provides it because:

  • it is no more difficult/easy to create a rest endpoint with vanilla meteor
  • as compared to other node rest frameworks
  • as compared to publications/methods with meteor

If that does not satisfy you, you can always meteor npm i -S insert-your-favorite-rest-related-package(s)-here.

As for auth, I had suggested you take a look at the source codes of the auth/token packages from simple:rest set of packages because they are very succint and easily applicable to any connect/express app.

In fact, if you take a closer look to any of the “file upload / download” packages in atmosphere, you’ll see that they all use rest endpoints embedded in the meteor app and the ones that integrate with meteor auth use the same (or extremely similar) ideas and apis that are offered by simple:rest and nimble:restivus. (Hint: you tap into the meteor login token on the client, pass it on either via some header or cookie and use it to validate/login on the server).

Oauth is even more fun! Because you can use meteor oauth packages and tap into the meteor account system, or use oauth directly just like you would have with any other rest api application.

5 Likes
11

Thanks for the clarification @serkandurusoy.

1 Like