Refactor/move file utilities with Meteor 1.3 Imports

core
1

So lets talk about improving the mess that is modules/imports.

My general take is that imports are 8 steps forward in terms of functionality and 5 steps backwards in terms of usability. That’s a net improvement of like 3 steps. But the overall traversal distance is 13 steps. And net 3 benefit versus 13 effort… leads to a lackluster ‘eh’ in terms of excitement in upgrading.

Now don’t get me wrong. I recognize that import statements are something of a gold-standard in programing languages, and for good reason. I had an entire career in C# where Import statements were de rigueur. So I’m not scared of Import statements.

But I also know that VisualStudio had Intellisense that could parse through all the available namespaces, and adding Imports amounted to selecting through drop-down menus. I also know that VisualStudio had refactoring tools that could move files, and that relevant path names would get updated. Which makes using the new imports that much more painful… because it’s obvious that there are missing tools for managing them.

So, what are the biggest pain points? After using imports for the past month and getting up to speed on how they’re implemented; the biggest pain-point is moving files. One of the major benefits of the pre-1.3 releases which autoimported into the global namespace, is that the resulting user experience was one where a dev could effortless rename files and move them around to work out the best application structure. One could effortlessly optimize naming schemas and organization systems.

With the current implementation of imports, however… once a few import statements get written referencing a file, it quickly gets entrenched and becomes difficult to move. Which leads to files not getting renamed to reflect content and purpose; of files not being moved to logical groupings; of file fragmentation. It’s a total mess. And let’s not get into the problem of missing modules and modules-not-found. We’ve traded the occasional intractable file-loading problem for an order-of-magnitude more missing-module errors. Which can be a blocker for junior devs or folks new to Meteor. (And one wonders why user experience and satisfaction is plummeting.)

So, what can be done about all this? Well, after a month of using imports, I’m convinced that we need file move/refactor utilities that can scan the file system, intelligently build up a map of all the files and import statements, and intelligently refactor the import statements. Just like VisualStudio used to have.

Questions are: do these utilities exist already? In NPM? In Webstorm? In Atom packages? If so, how do we incorporate them into existing documentation and builds? If they only exist as command line utilities, how do we incorporate them into the IDEs? If they don’t exist, can we marshal the resources to build these tools? Who’s interested in working on a utility for refactoring import statements?

Any thoughts or opinions or links would be much appreciated. I’m in import hell right now, and need a refactor/move utility badly.

EDIT:
All the relevant Atom package links that I can find or have been discussed in this thread:

And relevant GItHub issues:

8 Likes
2

I found this makes it less painful:

https://atom.io/packages/js-autoimport

However it will not refactor import statements if you move files, that would be cool.

2 Likes
3

I’m curious why the rest of the node community hasn’t run into this - I feel like it would have come up but I haven’t heard of anyone who is unhappy with imports there. What’s the missing link?

4 Likes
4

For the renaming of imports, WebStorm does this for you, all I did was set my language preferences to ES6.

3 Likes
5

They are still optional, right? So instead of complaining I think they’re simply not using them.

What I really liked with Meteor was the short time required to develop an application. With imports you need to multiply that time by 1.25, which is a bit sad.

6

I’m talking about the wider node community, where imports were never optional.

7

Oooooooh. Now that’s an excellent utility. Definitely going to add that to the Atom toolkit. It may already have half of the refactor/move functionality, since it’s already building up an internal map of all the import statements. Thank you, thank you for your report from-the-field that this is working.

I also just found this one:

That’s what I suspected. I’ll hand it to WebStorm… they have the better refactoring tools; and are doing their best to be the VisualStudio for JavaScript. The problem is that I know how to hack on Atom, but not on WebStorm. And I specifically moved away from WebStorm so that when I needed to hack on the editor itself, I could. I may renew my license though, just to check out it’s refactoring utilities for ES6

It has come up. The people who were unhappy with imports moved to a product called Meteor which managed to implement a Node based platform that didn’t involve require and import statements. :wink:

There’s massive selection bias and cognitive bias happening in looking to “the rest of the node community” on topics like this.

Exactly. Selection bias at work.

Regardless, I think this has come up before; and there are tools that exist to address this issue. I just don’t know which ones they are; and they haven’t made their way to Atom quite yet.

2 Likes
8

@sashko the problem with that train of thought is that is from the perspective of Node users, and not Meteor users.

Keep in mind that Meteor was built around NOT using import statements, and the methodology & part of what attracted users to Meteor was the fact that you did not have to spend time on worrying about those and could instead spent time on developing your app.

After C# and Django, using Meteor and NOT having to import, was a breath of fresh air. It’s one of the things I loved about working in Meteor.

So while Node users may be used to it and happy that way, Meteor users would likely not be. And considering some MDG employees have openly stated that they recognize Meteor has became more complex to work with, and will be receiving another pass at becoming more user friendly again, this is one of the top areas that should be addressed.

If a Meteor user from the beginning were to start with 1.3 onward, having to add imports to their older projects is easily the most jarring aspect of the new Meteor, and likely the least user friendly aspect of Meteor as well.

@Peppe_LG - the optional is only supposed to be temporary for backwards compatibility, and they said it will likely be required in the future.

JetBrains IDE’s other than WebStorm have support for automatically adding imports where needed , renaming files, etc. (although WebStorms Meteor support has not been upgraded with any of these features yet). It should not be very hard at all to add similar logic in Meteor to help assist with users adding imports to their projects.

I, for one, am dreading the day that I have to go through hundreds/thousands of files in my large project and convert them over to the new system. And I do not like increased development time on new projects. This is easily my biggest area of discontent with Meteor as it stands right now.

I truly believe if Meteor is going to make imports required, there should be a system in place that helps legacy users of Meteor, because if not they are basically being punished for starting earlier with Meteor & needing to put in a huge amount of development time just to be able to keep Meteor up to date.

Not to mention, one of MDG employees on here said something along the lines of “We have some things planned for this and we think you will be happy in the end” - so I truly do hope this area is addressed.

3 Likes
9

Webstorm is halfway there. After discovering last week that I did not know as much about imports as I thought I did, I began tinkering with file organization on an app I’m currently working on.

I must admit, I’m not happy with the results and miss working without them dearly. I know, it is optional etc, but one tends to try and move forward so that they don’t suddenly get left behind on a breaking change.

Anyway, webstorm can refactor moving files, but drag-drop and refactor > move yield different results, the latter being the better choice.

Furthermore, refactor > move does update other files that reference the moved one but the moved one does not update its own references, which is a huge gotcha!

Another gotcha is the problem of relative vs absolute paths. Absolute paths don’t get resolved properly so referencing across modules (read: top level directories that contextually divide the app, eg api/ui/startup/utils…) yield import paths that often look like '../../../../../api/etc' which make it much harder to understand the code. The everything is a global so I am using sensible namespaces approach was much better because it already carried a cognitive baseline to undertanding an app’s structure.

Also, you just simply can’t divide a file into two different files once it grows. You have to track down all the imports and update them manually.

Finally, re-exporting from imports, either directly using export from './something' or import { foo } from 'bar'; export {foo}; causes webstorm to loose the ability to actually resolve the contents of foo such that anything attached to foo becomes hidden away, whereas, directly importing it from its original export would yield in autocomplete assistance.

Anyway, from what I gather, one needs to really plan long and hard about the app structure and file naming schemes, otherwise, refactoring becomes a mess.

Now here’s what I wish we had:

  • Meteor’s familiar globals
  • A feature that allows wildcard adding of files withing packages (just like the root app)
  • Ability to use imports without having to place the imported file in the imports folder, such that it is merely a helper/override that mitigates edge-case load order issues.

If I at least knew imports would never ever become the default and old Meteor load order rules would stay there forever, I’d probably never switch to imports.

5 Likes
10

@awatson1978 Thanks for bringing this up! I’m currently migrating a relatively small codebase from the 1.2 client/server folder structure to imports and the api/ui strategy recommended in the guide, and have found it to be difficult also. (Can’t imagine trying the same on a large, mature project…)

To follow up on what you’ve said – re-organizing the folder structure in 1.2 was amazing. When you realized that you wanted to split a large file up, create subfolders for complex modules, etc., that was as easy as using a file explorer. Not so much anymore.

Also, I’m sure I’m not the only meteor dev who has experienced the frustration of wondering why an event wasn’t firing only to realize that the .js file containing the event wasn’t in the import chain.

That being said, the pain of folder refactoring has definitely lead me to think more concretely about naming/structure conventions. I’ve realized quickly that I’d rather pay now than pay later when it comes to reworking my import statements.

Moving a file and having to manually address the existing import references just feels like a stone age user experience, but ultimately I think it comes down to a question of ownership of a pain point.

Should MDG own this as a way to improve the meteor dev experience/migration path? Or should the community own it as a way to improve IDEs/toolsets?

4 Likes
11

Yeah, especially if you split up your app in multiple packages, it becomes very hard to move files around. And no editor knows what to do with ‘meteor/…’ imports. For packages you publish publicly it becomes almost impossible to refactor.

I do remember from my Java days that Eclipse had a shortcut to automatically organize all the imports which worked great, but that’s of course much easier with a statically typed language.

What I currently do is only use absolute paths everywhere, so at least I can do an easy find-and-replace.

2 Likes
12

I think @sashko isn’t necessarily just saying node, Meteor is following JavaScript as a whole. Which is fantastic. Aligning Meteor with mainstream JS is the most exciting thing to happen to Meteor since it’s release, now it’s finally a solid platform that I can bring other devs onto in a matter of minutes. Previously having to explain how all this “magic” happens was absolute hell, it doesn’t make sense.

I also disagree quite strongly with people saying imports make developing slower, while it may take you an extract 10 seconds to write your import statements, the long-term benefits knowing where things are to debug accordingly will save you far more time. Very much along the same lines as writing unit tests as you develop, takes longer, but will save you days in the end.

I do understand people are scared of change and 1.3 was a big one, but if you’re an experienced developer it was amazing. For the hobbyist or junior dev, maybe less so, but I don’t think MDG’s future is building a platform for them, it’s building a world-class platform that’s suited to larger projects.

If you structure your app well, then imports should be a very minor issue. Mantra examples have imports nailed down very well. Split your app up into ‘module’ folders, each with an index.js that gets referenced. Have a configs folder at the root of your app, which you can reference easily, so if you change something somewhere, you change it in one place.

Eg:

// configs/components.js

import MainLayout from '/client/modules/core/components/main_layout.jsx';
//                  or '../modules/core/components/main_layout.jsx';

export {
  MainLayout
};


// Rest of my app:
import { MainLayout } from '/client/configs/components';

This makes moving files around far easier, since you don’t have to update references app wide.

Then within each ‘Module’ you simply do

import ComponentName from './components/component_name';

Alternatively you could make an index.js in /components/ too and do

import {Component1, Component2, ComponentName} from './components';

This all depends on the style of app you’re building, but once you’re used to it, it’s a very simple and logical system. The ‘magic’ of auto things is great when first starting out, but unrealistic in production.

Anyway, just my thoughts on this topic :slight_smile:

Side note: I do understand the frustration of junior devs, I was one myself, as we all were. I do think a solid beginner friendly guide/tutorial should be made, whether this is from MDG or the community. However instead of doing the usual ‘LOOK HOW EASY THIS IS’ it should be a realistic guide, on how to build a good first app. Using imports, using correct folder structure. I personally never liked how the guides used the if (Meteor.isClient) {} stuff, I found that very unhelpful as it’s realistically not what you’re ever going to use in prod. Teaching beginners the wrong way isn’t going to help them.

5 Likes
13

I don’t think that anyone is saying that imports inherently make development slower. Rather, the way that imports are handled in Meteor, coupled with the fact that no IDEs really have out-of-the-box solutions for updating imports on file movements make restructuring a 1.3 app tougher than it should be.

To follow up on your junior dev point – I agree that teaching beginners the right way is something every tut should strive for. But learning Meteor from scratch was tough enough back when Discover Meteor was the go-to starting point. Knowing what triggers a template render, pub/subs, allow/deny vs. methods – a lot of these concepts are relatively new if you haven’t worked with reactive data before. Throw in modules and imports (and the fact that there really isn’t a de facto Discover Meteor 2 yet, and I think things are even tougher than before if you were to pick up Meteor today as a newbie.

2 Likes
14

Yeah, but it’s like we’ve all been eating bacon everyday for the past few years and now we need to start eating vegetables. We all know and agree that it’s better for us in the long term, but it’s very tempting to keep eating bacon. And I think this analogy holds for a lot of what Meteor is doing right now, like Apollo vs the current livequery system.

4 Likes
15

Mmm bacon. Good analogy.

Yeah as I said it’s harder for beginners, I know that. However for me, I had to get two other devs up to speed on my project, them knowing JavaScript, it was easy, they didn’t have to know Meteor. When 1.2 was around it took me a day to get someone up to speed with how everything worked and all the little details of Meteor, like the nested folders for build order. So I guess it’s an interesting one, experienced devs it’s probably easier than ever to start, beginners its harder than ever.

2 Likes
16

@joshig As you say it may not take long to write import statements, but keep in mind this is not Meteor’s traditional method. So what about the projects with thousands of files that do not have import statements? How much time do you think that will take? Might there be some issues with this in the future when backwards compatibility gets dropped, where developers would try to go as long as they possibly could before they do the upgrade, to avoid the lengthy job of upgrading a huge project to a new import system & folder format? What about when an update with vital security updates or something along those lines is released?

I think it’s great that they are building a world class platform. I do not think it’s great if they “sell the soul” of Meteor in order to become suited for larger projects. Meteor was built around fundamentals which are the reason most of us are here. Why should it even be an issue that some of us want Meteor’s future to be built around those same fundamentals as well? If you take those away, it’s not really Meteor anymore.

Especially on an issue such as this, where many projects (even my largest one) doesn’t actually require it and will see little to no gain from it? When adding support for “the old way” shouldn’t really be too difficult of an issue, as functionality similar to it is commonplace for plugin’s in some major IDE’s anyway.

And regarding getting devs up to speed… My job actually hired some frontend devs. They were somewhat familiar with javascript, but not specialized in javascript. This would have been completely sufficient pre-1.3 - and it’s a shame because I instructed them to put that as the requirement on applicant’s resume’s specifically because of Meteor’s state in 1.2…

But they started about a month after 1.3 dropped. The official tutorials also got much more difficult at the time, and even the folder structure was completely different from our huge project that they were going to assist me on. So I had to teach them Meteor (the way our work project uses it) from the ground up myself. It was a very unfortunate inconvenience that things changed that way after they were accepted for the position and right before they started.

I personally hope the day does not come where I have to go through the whole project and spend all the time restructuring it manually, for features that do not benefit the project. How do you think that will look to management in plans/progress reports? Definitely not time well-spent. So I am sure it will run in to that problem I mentioned earlier, where it will be pushed back as much as possible.

17

I do agree it’s a problem if you’re trying to update from 1.2 to 1.3, but my guess is from MDG’s standpoint, that’s a very short timeframe problem. It’s a bit like when everyone freaked out when Apple decided to change the iPhone cable, everyone panicked for a few months, but now a couple of years on, no one cares, but everyone is glad they did.

I honestly think clinging onto the past while JavaScript rockets forward is a mistake, and I’m glad Meteor is supporting the wider ecosystem. So yes, I feel for your situation, it does suck. I’m in a similar boat with some Angular 1.X apps that we want to migrate to 2.X. Or old react apps before the split from React to React + ReactDOM. That’s life, though, all those systems have improved massively, even if they do hurt a few projects in the short term.

I’d say if you’re happy with 1.2, stick with 1.2 for those past projects. Move to 1.3 for new ones. In production, you ideally shouldn’t jump to update everything for every release, unless there is a major new feature you need.

Just my thoughts. :slight_smile:

2 Likes
18

That would be fine short term, but long term I am less worried about new features (I’m pretty satisfied with features), but more worried about any critical updates to vulnerabilities/security/etc.

19

@joshig the main problem is not the imports, it is the refactoring hurdles that come with it. Yes properly structuring the code beforehand and creating module based facades are two most basic things that should be implemented anyway. But, every app comes to a point (even within days/weeks) where names should be changed, files should be moved or split and javascript editor/ide offerings are far less than complete in terms of refactoring support. Even reportedly successful refactorings turn out to miss a few things, thus introducing subtle bugs or more overhead.

But yeah, this certainly is a move forward, but javascript’s (and Meteor’s) “move” does not always feel that forward. Or it does not feel well executed.

In fact, javascript/meteor move feels so sporadic and backwards-problematic that I strongly disagree with this statement:

In production, you ideally shouldn’t jump to update everything for every release, unless there is a major new feature you need.

Because you never know if two minor versions ahead, you’ll be left so behind that it would become impossible to even publish hotfixes due to any one of javascript core, meteor core and any library/package out there having abbandoned support for your deployed version. So you are kind of forced to keep up and even ahead.

2 Likes
20

Thank you for this excellent breakdown of the current state of refactoring in Webstorm. These are exactly the type of issues I was hoping to discuss in this thread.

Suffice it to say that Atom is basically in a similar state, if not even a bit further behind. Atom has a quick-move package, which doesn’t look at imports or file content at all. It has a refactor package which is a glorified version of ctrl-D multiselect, but doesn’t handle files. There’s no drag-and-drop. Files can’t be quickly divided. Not sure if moving a file would trigger a reload in the jsimport package, but the cynic in me guesses that it wont.

The bright side, however, is that with Atom we have the source code to all these packages (and more). So we know how to build up the imports map using jsimport. We know how to create the IDE menu selections using quick-move. We know how to search for import statements using refactor. Getting a working file-move refactor utility in Atom is doable by the community.

Excellent questions. The cynic and realist in me expects it will fall to the community.

Although I would like to state for the record, that there is a fairly large group of experienced devs in the Meteor early-adopter community, who view Meteor as the most sophisticated refactoring tool in the Node ecosystem. Meteor has historically attracted junior developers because it let them build Node apps without having to manage Node require statements; and moving files around and the build system watching and rebuilding automatically amounted to magic. However, that same build system has also attracted senior developers because they see it as an entire refactoring pipeline with filewatchers, file scanners, autoimports, file writing, etc. And that refactoring pipeline has to be rebuilt as of Meteor 1.3 to accommodate ES6 import statements.

So this is something that MDG has historically owned. Whether or not the choose to continue owning it… I got no clue. But this isn’t something new or some type of feature request. This is very much a core feature of the build system that’s been around since 0.3 or 0.4 days that’s possibly being removed, if the guide and statements about imports being the default in future releases are any indication.

[quote=“seba, post:11, topic:25176, full:true”]
Yeah, especially if you split up your app in multiple packages, it becomes very hard to move files around. And no editor knows what to do with ‘meteor/…’ imports. For packages you publish publicly it becomes almost impossible to refactor.[/quote]

Another great consideration. We’re definitely getting enough to create a bullet-point list of recommendations and an action-list. Will need to add meteor/.. imports to that list.

This should be a recommended best-practice in the guide and in example applications. Thank you for pointing this out. There’s a world of difference between absolute and relative paths in import statements; and I’m beginning to get to the point where I can’t see anything good about relative import paths.

# GOOD
api/ui/startup/utils

# BAD
./style

# REALLY BAD
../../../../../api/etc

I know for a fact that the Meteor Chef Base app is currently shipping using relative paths. That’s how ./style statements crept into my current app; and it’s slowly-but-steadily spiraled out of control into import-hell.

And looking outside the Node community, the C# and .Net community uses qualified namespaces, which are basically absolute paths.

Imports System.Text
Imports System.IO  
Imports Microsoft.VisualBasic.ControlChars  
Imports strbld = System.Text.StringBuilder
Imports dirinf = System.IO.DirectoryInfo

So, can anybody think of a situation where relative paths are essential? Meteor packages? Npm packages? Otherwise, I think there’s a case to be made to update the Meteor Guide and various styleguides and best-practices to use absolute paths in import statements.

4 Likes