Meteor, the loose sand development stack?

[mro]

Hi guys. New to the forum, yes, not new to Meteor. Have been a Meteor user since version 0.who knows, and created several applications with the platform. Whilst not being a professional developer, I do use Meteor to create applications from time to time, and I always enjoyed using it.
However, since the start of using it, I have always experienced frustration in the updating process of a Meteor app. It never seemed to end well. Over time, I have felt Meteor and all of the packages combined into your applications, to be like loose sand slipping through your fingers. I have never felt it to be a coherent-working-together-symbiont-organism. It always felt like you had to “juggle” the parts of your app with increasing difficulty to keep it all work from the get go. Every new update was very often a source of frustration since it was more common than not that it would break your application.

For incidental developers this can be reason enough to stop using the platform. Since you do not have an extensive documentation on your application, and a deep knowledge of all the packages you use and their co-relations. Add to this the abacadabra error messages you get after the updating of your Meteor project instance breaks your app, and there it is: the Monster of Frustration !
Now, I have an app created with version 1.8.1. working just fine as it is, doing it’s job, not harassing anybody…

I just ran an update of the Meteor project instance to the latest version 2.3.4. and, yeah, sure enough, the app broke down and humbojumbo appeared by droves in the terminal. Motivation to throw the whole thing out of the window, knowing that analyzing this would take me days and days, needing to get into the app deeply to understand what packages I had installed, and check whether they were broken or not, and if so, what could be done about them.

I just HATE that system. This is not Up-Dating, it is Up-Breaking. I just had expected that by now Meteor would have found a way to avoid apps getting broken up by updating to the latest version. Turns out, it still hasn’t. That’s disappointing. The Meteor Platform still looks like loose sand to me.
Is this EVER gonna change?

For now, I tackled this by deleting the updated package, and replacing it with the backup I made just before updating, and sure enough, it worked again. But, this is frustrating as hell.

[truedon]

First before updating you need to read the breaking changes so that error crap makes sense Meteor Changelog | Meteor API Docs

As a developer we face this all the time, I hear your pain as even for someone who’s studied to work in this, it’s really annoying and happens everywhere in computing.

Updating is always pretty annoying like this on Linux and many times I have had to just reinstall the distribution, even if you update regularly there can be problems when you go across major versions.

It’s just how packages work, in python and rpm’s for ubuntu it is also really annoying and the errors often much more cryptic.

The best way is to read the error message and walk back try to find the plain english meaning, look for filenames and Google it.

Meteor itself is pretty solid in regards to this and that’s why I prefer to use it as it has it’s own core package set and they do publish a changelog and breaking changes. I’ve found that NPM packages and using an entirely roll your own setup is even more of a PITA often as you have to deal with undocumented issues that are only applicable to your use case.

I understand you hate the system, because I hate it too. I wish updates would be painless and just come in. The problem is breaking changes and up-breaking is a good term for it. There isn’t a better design yet for updating software really. I’d be game to try a new paradigm but this is the best we can design it really.

To make it as painless as possible, except you are about to brik it by updating. Make a copy, update there. Check small details like version of node and npm. Don’t be surprised if a wall of error text appears. Try to read it and figure out what the causes are. Look at the changelog and if it’s still making no sense at all hire a developer to fix it for you so you can get on with running your business and not wasting hours wading through this crap. Believe it or not but this is the majority of an admins job. I have been given broken updates so many times across so many distributions and I just got it fixed because I was being paid to do it. If you don’t update frequently it will make it even more painful btw. So try to update regularly and it will just be a smaller issue.

[mro]

Thank you, Truedon, for your empathic reaction. I appreciate it, and I appreciate your suggestions and tips in order to make life more easy. Your answer is very well put together, and it serves me well. However. Allow me to make a few observations in this regard:

I think this is “failure by design”. And we simply shouldn’t accept it. We should strive for more than simply saying something like “well yeah, but this is the way it is with all software development packages” and next just continue to accept needless work and efforts as something that comes with the trade and cannot be changed. If that is the design of things, the design is wrong (suboptimal, I think sounds more friendly) and needs improvement.

How is it, that we accept that when a software development platform produces a new version, the customer layer of that platform simply gets a crash with a ton of gobbledygook and next have to check their products intensively spending hours and hours to just find out WHAT just happened and maybe needs repairs?

It is just for that reason (and my experiences over the last decades with software) that I developed the mindset: “if it still works, don’t update it!” or just simply: “don’t repair what isn’t broken down”. Keep your hands of it, and feel grateful it still works. However, sometimes this philosophy doesn’t fly, in this case because deploying to Galaxy requires Meteor apps to be version 2 or higher. Yeah, well, thanks for that… and then of course you have security concerns to deal with as well with increasing priority.

Why is it, that when a group of developers creates a tool that is expected to be upgraded and updated over the coming years, they do not also create a system that allows for that to happen smoothly, without disrupting the products made by that tool? I just makes no sense. There really is no excuse for not having implemented safeguard technology that would prevent you from update-crashing your application just because you have a package included that has some issue.

Now I get it, things quickly go way more complicated than it just being a package that isn’t certified for the updated Meteor version. I know that. Because who knows how that specific package is used or abused in the application that stops that application from working after updating. I get it.

It might come down to a choice to keep the balance somewhere between limiting the freedom of “using a package any way you want” on the one side, and having a more robust upgrading and updating procedure on the other hand. Things are not simple, I get that. But like with more decisions, there is a cost and a consequence involved with all design decisions.

Of course, you are right that Meteor is not the only product that lives with this kind of problems. I have seen platforms that would require you to first update through ALL of the steps in between instead of just simply updating an app from version 1 to 6. You had to go through a whole sequence of misery, updating from 1 to 1.1.1 via 1.1.6 through 3.4 and so on. You get it. But I also know, that some platforms have put way more significance into debugging those steps relating to updating, because they in the end got tired of the hurt they put on to their customers and the feedback they got from them.

Choices, choices.
Don’t repair what isn’t broken down. The Golden Rule. Until it no longer is…

[truedon]

Yes I mean the standard is to not make breaking changes but it just isn’t always possible when you are doing speed optimization or adding new features etc. The problem with web tech is its evolving and so that’s why we have package managers.

All operating systems and packages follow this same methodology, it’s just some projects and teams follow it closer then others. It’s the cost of having a entirely free code base. So just have to work with it and get it done. Sometimes that means a bit of pain. But in the long term, the ability to use others code and collaborate with open source packages and free tools etc far outweighs update problems.

A really valid point here is - If you don’t need to update, don’t. If everything is working fine then just run it. I know people with software from twenty years ago, it still works, it does the job so they just don’t care about updating. As long as it’s making profit and doing what it was designed to do, who cares.

[mraju]

FWIW, I don’t think the Meteor developers themselves are at fault here. I actually think they bend over backwards to maintain compatibility in the core packages. And whenever I get an error from an upgrade, I usually find the change listed in the breaking changes section.

The main issue I find is that meteor’s 3rd party packages are all severely out of date. At this point, if you use anything besides meteor core packages and npms, you are SOL for any type of updates. And even the npms, if they’re primarily for meteor, often haven’t been updated in years.

Personally, this is where the source of most of my breakages have come from. If there was a healthy ecosystem, then package maintainers would upgrade their packages quickly with every new core release, and we’d just have to do one update for all the packages and core stuff to work again.

Right now, for example, it’s almost not worth updating meteor in my oldest application because the vast majority of packages are being held back due to dependencies from old, unmaintained 3rd party packages. For example, I use the brettle:accounts-anonymous packages, which depend on a specific version of meteor’s useraccounts system. Unfortunately, it hasn’t been updated in years, so I’m stuck using the old user-accounts version, unless I take the time to take over maintenance myself, find a new package to replace it, or remove the functionality it provides. None of them are great options, so I’m stuck with the old user-accounts package until I “find time” to implement a better solution (hah!).

Replicate this over a bunch of random parts of the codebase, and you have a recipe for exactly the hodgepodge of versions that you mention. But personally, I think it’s more due to the decaying of Meteor’s 3rd party ecosystem than the core developers themselves.

[philipmjohnson]

I am a professional developer who has been using Meteor for many years now. I have had a completely different experience than @mro. I’ve found Meteor to be one of the most backward-compatible tech stacks I’ve ever used. Here are a few of the systems I’ve been maintaining throughout many versions of Meteor:

• GitHub - radgrad/radgrad2: Source code for RadGrad, Version 2
• GitHub - bowfolios/bowfolios: A sample Meteor application implementing a portfolio and project portal.
• GitHub - ics-software-engineering/meteor-application-template-react: Template for application development using Meteor, Semantic UI, and React
• GitHub - ics-software-engineering/meteor-example-form-react: Example of form controls using Uniforms and Meteor

I suspect the reason why my experience is so different from @mro is because I have exactly the opposite methodology around updates: I update all of my applications to each release soon after the release comes out. As a result, most of the time there are no problems, and when there are problems, it is usually a single problem that is very easy to find and fix.

I think it’s useful to think of your tech stack the same way you think about team development. Consider a situation where you have a group of developers who develop new features in branches and commit to the main branch when their feature is finished. The way to stay sane in this situation is to update your branch from main each time a commit is made to the main branch. If you do this, then you minimize the number of merge problems you have to deal with at any single point in time, and you also can track down problems more easily because you know the single commit to the main branch that caused the problem.

It is a recipe for extreme frustration (and maybe an abandoned feature!) to instead decide to “never update from main” during feature development (because that appears to avoid encountering merge conflicts). In reality, this doesn’t avoid the merge conflicts, it just means that when you eventually have to merge, you’re going to get all of those problems at once, and it is going to be way, way harder to figure out how to resolve them.

So, @mro, I totally understand the temptation to avoid updates: “If it’s not broken, why fix it?” But to update from 1.8.1 to 2.3.4 is an update of 20 releases at once! That’s just crazy. At the very least, you will find your experience to be much improved if you update each time a minor (i.e. non-bugfix) release comes out. That would be 1.9, 1.10, 1.11, 1.12, 2.0, 2.1, 2.2, and 2.3. That’s 8 releases over the past 20 months, which is certainly not a huge development burden.

Good luck to you!

[mro]

Dependency Hell

It’s a struggle, isn’t it? Choices, choices, choices. Do I update, or do I not? Is there a “good answer” ?
I don’t think so. It is more or less depending on the priority of your app, how you use it, and in what setting the app is deployed. There is no such thing as an easy answer.

“Always update!”… well, heck NO !! Think before you leap. A small app in the sideline which has been built with a tiny budget and performs well as it is, why should you if you do not have severe, high priority reasons to?

“Leave it as it is!”… well, that remains to be seen! If your app is your company’s lifeline, that is way too simple thinking. In that case, one would rather advise multiple environments (developing-testing-production) and a controlled release & update policy to stay ahead of threats to your precious app.

I think every professional senior developer or IT Manager knows that quite well. Doesn’t it all depend heavily on budget and manpower, which again depends on priority for the organization?

I agree with Philipmjohnson that updating with a gap of 20 releases is crazy in the way things are organized right now with Meteor. I also don’t think it can be done AT ALL as it is now without expecting major repairs in what not. And therein lies my problem exactly, Philip. You should be able to do that, in a perfect world. The current way things are organized however, simply does not allow this without major work.

Dependency Hell is maybe not the best way to manage an important app. Too much can hang on it. You would need to invest way more in terms of platform hardware, environments and manpower to keep such an app up compared to using an IDE that “takes care of that” for you. Yes, I know: there is no platform without it’s own problems. Problems sort of go with the territory of software application development and deployment. It’s all in a days work. But I would make my point, that choice of platform (and subsequential issues) depend on budget (which again depends on…).

I worked intensively with low coding environments, and in those platforms (where you quickly realize you can’t really do shit without coding, although they like you to believe otherwise) the company that provides the platform tries to keep you away from all those dependency problems. With different levels of success, and I have seen multiple instances where they couldn’t keep up either, and apps broke. Mainly non-mainstream apps that had custom coding inserted broke regularly. There is always a point where the same issue occurs: Dependency Hell.

Once you choose a package for integration in your app, you choose your Partner in Dependency Hell, and you are more or less chained to it.

Which means, that you should be budgeting for application maintenance and you should be budgeting for re-design at some point in your applife expectancy. There is no such thing as a 1-time budget for any app, unless you expect it to only serve a year or so. And even then: good luck with that.

It is a task of IT Managers (which I must sadly report to be one…) to sell that idea successfully to the C-level Officers who “just want you to get it done”. It is easy to comply to that idea. It is more difficult to bring up the unpopular talk about the budget needing to be double that what the C-Level Officer has in mind most of the time. It should be done. It should not be avoided, if you as an IT-Manager want to keep your developers out of the wind in the nearby future.

[vikr00001]

As has been noted, this is true of a lot of tech stacks. I recently updated an important non-Meteor npm package. This npm package is very widely used inside and outside of the Meteor ecosystem. With the update a lot of objects it returns are now immutable. There was a lot of code in my app that needed to be changed. I had to go through the app, using lodash clone and cloneDeep to make the objects mutable.

It appears to me that the awesome packages we use are going to be improved and iterated, and this is a good thing. We of course have to update our apps to keep up, if we want to use the new features.

[MastaBaba]

I’m just chiming in to say that I’ve many times faced similar problems to what @mro describes, and I do not skip 20 Meteor updates. But, if I can avoid it, I also don’t move with every update, of Meteor, or installed packages, because when there’s failure, and there too often is, I know I have to put in hours to fix something that wasn’t broken.

I understand this is a ‘feature’ of package-based solutions, and why I try to avoid them, if I can.

Case in point, several times now, I relied on a package that stopped being maintained and stopped working with some updated versions of Meteor, which meant I had no choice but to replicate the bits of functionality I needed, myself.
For apps that are not, or not meaningfully, revenue-generating, this just puts me off using Meteor; there’s always a risk I have to put in time I don’t have to get something to work that broke because of something I have no control over.

[illustreets]

@mro

“ Always update! ”… well, heck NO !! Think before you leap. A small app in the sideline which has been built with a tiny budget and performs well as it is, why should you if you do not have severe, high priority reasons to?

Well, if you do not want to update, then there is no issue to complain about, right? Should just keep it running for as long as you can and hopefully it will be chugging along uninterrupted for a long time. I don’t see how Meteor itself forces you to update Node.js on the other hand may be a different story, and may even fall in the severe category.

I understand the pain causing the above diatribe, but the criticism is wholly unfair, and considering the significant effort invested by the new owner and the community since the recent acquisition, calling it “loose sand” is just not representative of reality. In the ever shifting landscape of JavaScript & Node.js, Meteor is quite the contrary: a rare, demonstrable example of stability and backwards compatibility.

Just let’s revisit the Angular 2 vs 1 story a little, for perspective, or even the early React changes that forced developers to keep rewriting. Then, for those familiar, there is the deprecation of request and other famous similar problems. How about some widely used package being simply removed from NPM by its author, breaking countless packages in the process. Dependencies break all the time in the wider Node.js ecosystem in a far worse way that Meteor itself (never?) or its packages ever did. At least with something like request, if you use Meteor for development, you hardly felt anything changing under your feet, while the community quietly updated the packages who used to depend on it.

I have no knowledge of how your app is built, but I also have been using Meteor since early days, version 0.4x (the spring of 2013 if I remember correctly), and built a fair share with it. I only remember spending one - two hours at most, in the rare moments when something broke, even with major updates. In our main product we use 129 packages by external contributors, and 88 developed by us internally. But we monitor those packages, and we make sure they work. We refactor, test, and re-deploy as a continuous process. For better or worse, as developers this is the work we ought to do.

I swear I cannot relate to the problem you describe.

This sounds like a classic case of software rot, indeed, in some places caused by outdated packages. But forking and/or modernising some of those packages should be relatively easy, and in some cases even imperative, as a matter of good housekeeping. The package brettle:accounts-anonymous for instance, contains a grand total of 74 lines of code, except for tests. That, if you ask me, should not really be an external dependency, particularly if essential to the correct functioning of the app.

[mro]

Hi Illustreets. Thanks for responding!
I can appreciate your stance, and your opinion, to which you, of course, are fully entitled.

However: “calling it “loose sand” is just not representative of reality”… did you miss my question mark?

It is always good to discuss stuff, only in this way one can expect to make improvements. If everyone just applauds the Meteor developers and their products, what then boosts your chances of getting stuff improved?

As a former national champion in some sport, I know that you gain almost nothing in your performance from continuous praises by fans. You gain performance, knowledge, ideas and so on from honest critics from those who dare to vent it to you, and the discussions that follow. Of course, not every point of critic bears merit, and it could be mine doesn’t. Yet, it is worth the discussion since, as it seems, I am not the only one having difficulty with Dependence Hell. And yes, it is not limited to Meteor by far, nor did I state so. It is a flaw by design. We simply don’t see a better solution yet, but that doesn’t mean the current solution is perfect, and it doesn’t mean some creative genius can’t stand up at any time to sprout a better one.

[mro]

Thanks, MastaBaba, for your reaction. Maybe until a better design is found, it would be a prudent approach to not install too many, too tiny, packages that one could also develop yourself? Leaves the larger, more influential ones, for which the issue still stands. It is a struggle. The emergence bandage for the problem for now seems to budget for it in advance. But hey, there are the Boundaries of the Market as well, one has to be competitive in price and costs. With the real picture you might be outpricing yourself. This is not an easy problem.

[mro]

@ vikr00001: True. It is a weak point of stacks design. No matter what platform lies beneath it. It still can be quite frustrating to see it show it’s ugly head once every while in your favorite development platform.

[macrozone]

I always try to observe and judge the activity of any library i am using and try to avoid dependencies that do not seem to be maintained. I also try to avoid packages that do not seem to do much.

Personallly i think the dependency hell situation has improved a bit over the years, at least in npm libraries, as many have adopted semantic versioning, which gives developers more transparancy on what has been changed (patch version = bugfix, minor version = new feature, major version = breaking change).

Its a bit trickier for platforms like meteor though. In particular because it is both a library and a runtime. So it does not follow semantic versioning, every version can break something or nothing. On the other hand, packages still assume semantics of the version numbers by defining constraints. Meteor jumped to 2.0 without any (obvious) breaking change. That lead to the situation that many packages complain that they are incompatible to meteor 2.0, altough actually they were. Whereas meteor 2.3. updated to node 14, which might actually break some packages, but the version did not indicate that.

Probably meteor should follow semantic versioning more closely, at least for its packages.

[vooteles]

In my opinion the source of issues with long term app maintenance is not Meteor itself, but rather the ease with which devs add atmosphere/npm packages into their apps. Meteor has maintained backwards compatibility for years and when breaking changes are introduced, they are usually well documented in the update notes. As such, I don’t think Meteor itself deserves being called a “loose sand development stack.” Quite the opposite.

However, I think it is really not reasonable to expect the same kind of stability from code bases that are infused with 5, 10, sometimes 20 different packages from atmosphere or npm, all with different levels of maintenance and usually some already abandoned for years. I personally try to set myself a very high bar when considering adding any new packages and I only do so when I see that the benefits from a package really outwheigh any maintenace cost. And of course I usually prefer packages with known reliable maintainers and also packages that have a limited API surface.

This has served me really well. I don’t recall having any trouble upgrading Meteor versions over the years.

[rjdavid]

@macrozone

Probably meteor should follow semantic versioning more closely, at least for its packages

When I started with Meteor, I had difficulties with this initially. It was such a pain (in terms of expectations) that Meteor was not following available standards in this regard. Recently, seems like that both patch and minor version movements are according to semver. The major version change for Meteor still means a “major change” instead of just breaking changes.

[filipenevola]

I’m using Meteor for years so I know how important is to keep things up-to-date.

Now that I’m working with Meteor directly we are also keeping this in mind all the time and avoiding breaking changes as possible.

For this kind of discussion I would like that us, as a community, be very specific.

If something is not working for you it’s better to be specific instead of generalizing.

For example, which change specifically did we make that caused problems for you?

Of course, if you are trying to migrate from multiple versions you need to follow our migration steps and breaking changes section in the changelog. There is no other way around.

We always keep this changelog up-to-date, even when something comes up after the release.

[filipenevola]

@truedon

That is until they add a broken feature and it’s force pushed to everyone.

Hi @truedon, I don’t know which broken feature we’ve added, could you be more specific?

And as I said above, generic comments don’t help us on having a better platform for all.

I believe the goal of everybody here in the Forums is to have a better Meteor everyday so we should try to focus in this direction on every topic and message.

[filipenevola]

I forgot to mention another thing.

Even now that I’m working in Meteor I still have many Meteor apps that I own in production.

And of course, Meteor as a company also has many Meteor apps so we don’t have any intention of breaking things for the sake of breaking things.

I’m really proud of Meteor history of backward compatibility. This is a great selling point of Meteor and that is why I would like to have more concrete examples in this discussion so we can improve

[truedon]

Discarding livedata message