You have to have a parser for SQL update queries. Then you can know which fields are modified in base tables. And then you have to be able to propagate changes to those fields through joins, aggregations, views, materialized views, and custom functions which can be used in SQL queries. Not to mention that updates can come from triggers, so you also have side-effects your SQL update queries can make.
So, unless you want to require users to manually specify how their SQL update queries propagate, it is not so easy. But would love to discuss this more somewhere else, if you are interested (to keep this thread on topic). I have been looking into this recently quite a lot.
You can limit SQL to just a subset, to make it more or less allow only what MongoDB allows, but I think one of main reasons why people want SQL support is to integrate with legacy apps, and to have features MongoDB does not have, and not just to have a different query language for MongoDB-like database.
For several years already, vlasky:mysql has provided a reactive MySQL integration with Meteor. It is stable, perfomant and has been used in a number of enterprise Webapps.
Reactive queries are triggered by following the MySQL binary log. I have posted extensively about it on the forums.
@awilkinson Tiny should also feel free to let us all know how we can help during the transition and beyond to help make Meteor a successful investment for you guys and an amazing free to use tool for all of us. I am pretty new to the scene but I really enjoy spending my time in the forums, building hobby apps and growing my knowledge of Meteor, Vue, Node, Mongo as well as what is trending in the web development world. It would be sad to see that all disappear and I am sure I am not alone in feeling that way
Where I work, there’s concern about ecosystem lock-in. Using Meteor feels like building a Meteor app, not a Node app that uses Meteor as its framework. We already use Vue as our frontend, so Meteor is really only supplying the data layer and build tool; I tell people that it’s more appropriate to say it’s a Vue/Node app as Meteor-specific code makes up < 10% of the app. But still, the perception persists.
I think Meteor is an excellent choice for companies building internal apps, and I suspect that that’s a large segment of its usage. There’s no concern about scaling (though I disagree with the folks who still argue that Meteor can’t scale) and speed of development is often much more important than limitless flexibility. What would make Meteor an easier sell where I work would be:
Finish the job of splitting out all of Meteor’s parts into NPM packages that can be used with or without the monolith. This was already started with accounts-js, I think the next biggest target is LiveQuery, possibly depending on Apollo (discussed here). Then I truly can say I’m building a Vue/Node app, just adding LiveQuery the same way today I could just add Apollo.
Decouple the data layer from the databases, and support multiple databases with reactivity. So just as the client-side has a primary framework (Blaze) but other supported frameworks (React, Angular, Vue), the data layer could have a primary database (Mongo) but other supported databases (MySQL via binlog tailing, as shown by vlasky:mysql; Redis via oplog tailing, as shown by cultofcoders:redis-oplog; Postgres via triggers/listeners/other magic as shown by tozd/node-reactive-postgres). Supporting SQL databases is important for enterprise customers not just because of the mindshare that SQL still commands but also because hosting SQL databases is cheap and maintenance-free via Amazon’s RDS (Relational Database Service). Amazon supports RDS-like Mongo 3.6 via a service they call DocumentDB, but it’s very expensive (at least $200/mo) and therefore inappropriate for internal apps with only a handful of users.
Both of these suggestions have been on the Meteor roadmap for years. SQL support has been on the to-do list since the earliest days of development, back when it was a Trello board. Moving away from Atmosphere into NPM has been in progress for the last few years but progress has slowed as resources have dwindled. But both efforts are still good ideas, and completing them would breathe new life into the framework by making Meteor an option for those who currently wouldn’t consider it, and by making Meteor (or parts of it) an option for adding onto an existing Node app.
I find Mongo, DDP, Pub/Sub to be sleek and modern, whereas SQL DBs to be overly bloated and archaic. Reactivity was the principal feature to embrace Meteor. If I wanted to make a static site with MySQL or PostgreSQL, I would choose PHP or RoR. Or .NET for the “true Enterprise” (meh). Reading all this hype about Apollo is disappointing. Maybe instead it would be better to support more NoSQL DBs and/or better integration with alternative reactivity functionality like Redis oplog?
I agree, but sometimes other considerations dictate the architecture. Note that both MySQL and Postgres now support a json field type, so they’re inching closer to the conveniences of Mongo; and both can be made reactive. There’s a great opportunity here for a library to smooth over the rough edges.
From my observation dealing with Postgres at work (including all the tools available) and having own Mongo-based projects, it is like comparing Linux to Mac. While I respect Linux a lot, sometimes (most times?) I just want to be efficient and do not have time to build the binary from sources. This is exactly why I like DDP and Pub/Sub - it works, great, out of the box, and is heavily configurable. What else is needed? I would rather prefer to build great apps than port existing ones to every new hyped framework every 4 years…
No, unfortunately. The closest ones that I’m are of are the ones I cited in my post. The MySQL and Redis ones have been used in production for years by various people.
meteor’s initial idea of reflecting a subset of the servers database on the client was revolutionary and simple. But it forces you to have one fixed Business-Model throughout your application, from client to database. This can get problematic in bigger applications, e.g. if you have some computed data or need to map your data to another structure because it can be more easily consumed. Usually in meteor, - to not loose reactivity - you tend then to create new collections, where you copy data over, or you create “virtual” collections client side and a crafted publish where you implement .added.removed etc. which can get a bit hacky and hard to understand (and in case of copy data over also error prone).
Another problem is, is that your client side collections are “global” to the client, but their contents depend on which subscriptions currently are active. In component driven development, this can lead to problems, because you try to co-locate the data-fetching with the component where you need the data. You need then a strong ceremony of always subscribe to exact the same data as you fetch in your component and you also need to be sure, that the subscription you are using uses the same selectors as your publish function. E.g. its easy to accidentially subscribe to a wrong publish (or forget it) and it still works, because coincidentally another component is mounted with the subscription that you need. If that other subscription now unmounts, you suddenly haven’t got the data anymore.
apollo-client solves these problems by allowing you to describe the data that you need in your component exactly and isolated.
Also in my experience it allows you to model your Business-Model exactly as you need it and gives every developer involved (backend or frontend or even third-party consumers of your api) a common ground to discuss and understand your application. It even helps, if you are the only developer, because it removes cognitive load from you, you can always look at the graphql schema to understand the business model of your application.
Thank you for the detailed explanation. I have got more information from your post than from many previous posts and/or marketing materials of MDG on Apollo and/or GraphQL. Which also shows that MDG should improve communication in this area as well.
There are 2 points regarding this:
Microservices - how many of actual Meteor devs do use it in Microservices architecture? I have setup some of my apps into modules to be able to interact with each other over ddp, but that is rather simple and is not “Kubernetes orchestrated” set of instances, rather than manually setup 2-4 Meteor servers…
Reactivity across multiple microservices - is it really required for every microservice? or it is better to make a single “sync” service and use only it, to pass the data to clients over ddp? then this could be achieved with an instance of meteor and multiple other REST(Express)-based instances etc…
Now if your app is that complex to have many services, then you most likely have at least 10 devs and at least a single dedicated devop, who can setup (and maintain!) whatever system is needed. Then if you do think that maintaining the whole setup worth the benefit, that you get from it, then it is must be time to move forward and rebuild the whole system without meteor. Or if you know you have to have that complexity from the beginning, then it does not make sense to start it with Meteor (but good luck with building and maintaining the whole thing, even in large teams with lots of resources). At the same time you can also have some other separate systems run on Meteor for their dedicated tasks.
we orchestrate meteor and other node apps using kubernetes, but that’s not really microservice approach, its just scaling it up to as many apps as you need (usually with a 3-app mongodb-replicaset). Its the same as deploying any other node app.
“Reactivity” with pub/sub works across multiple instances as well when you use either the default mongodb-oplog or with the help of https://github.com/cult-of-coders/redis-oplog. Default mongodb-oplog adds more load to the mongodb, therefore i would add redis-oplog (which needs an additional redis, but thats very easy to install on kubernetes, if you use helm or similar)
meteor is perfectly suitable to be deployed on kubernetes or similar. We also have it sometimes as pure api-app without any frontend, but then, it could also be just replaced with any other packager.
I also want a lot of the things mentioned here (COUGH-hmr-COUGH), but I believe the single most important thing is to hire the right people (to follow up on @sacha’s excellent recommendation to 2x Ben’s salary). Hire some/all of the most dedicated developers in the community and let them get together and figure out what is best.
My advice to Tiny, don’t even wait for their resume’s, hunt them down and give them all the money they want, and watch the community rally around them while they build the most amazing future for Meteor
Which also shows that MDG should improve communication in this area as well.