Package structure for circular dependencies


#1

I’m trying to design a package based layout for a new app.

The problem is interdependent collections, their publications and helpers (transforms).

I’ll be using the usual suspects:

  • aldeed:collection2
  • dburles:collection-helpers
  • matb33:collection-hooks
  • reywood:publish-composite

So let’s discuss this over an arbitrary example. Say we have two collections:

  • Posts
  • Comments

For sake of argument, let’s say I’ll create the following helpers and respective publications:

  • Posts.latestComments
  • Comments.postAuthorFirstName

such that the collection helpers and the publications (with joined cursors) kind of inter-depend on each other.

So if I were to create separate packages for Posts and Comments, I would then need to api.use Comments from within Posts and vice versa.

But that’s a circular dependency which screams problem in all kinds of ways.

So what would be a clean package layout?

Separate collection helpers and publications into their own packages? That also sounds bad to me since my code will be scattered all around. Normally, I’d keep the collection, schema and collection helper definitions in the same file.

So, what are some novel ways have you come up with to structure your apps in such interdependency situations?


#2

I think that this is a mistake.
In traditionnal “entreprise” OO, you have a layer for the database, a layer for the models (Classes), and an adapter between the two. Your classes shouldn’t depend on the database, which they should’nt even know about. That way if you choose to change your database, your models doesn’t change. It prevent doing too much binding (technically, and designing them) between your domain model and the actual data format.

Could you provide us with a more detailed example of what you are trying to do, i’m not so familial with joined cursors, i think it’s better to relate objects to each other once they have been brought of the database (instantiated as “real” objects).


#3

I have a similar layout in terms of commenting. I decided that comments are not actually dependant on posts since I want to be able to comment on other things. So my Posts package uses the Comments package api and stores the id of the attached comment. Comments on the other hand have no dependencies, but when they are created they take an optional parentId which allows me to store the id of whatever parent document they are commenting on.

P.S. comments in my format are not a single user comment, but a list of all the comments attached to some document.


#4

Please jump ahead if you are not interested in reading some philosophical BS I could not let go without saying :smile:

Ramblings

Well, an argument around ORM, abstractions, domain modelling and OO (needless to say in a fundamentally functional world of javascript on top of an essentially non-relational nosql world) is a rising discussion, even among the likes of Martin Fowler and Joel Spolsky.

So I’m not going to go there :smile:

But (there always is a but), coming from the java (ee) world with ages of experience in database and domain abstractions like ejb (yikes!), ibatis, ebean, hibernate, jpa and patterns around daos, interfaces, implementations and layers upon layers, I’ve come to take refuge in javascript’s simplicity which we once had with fortran, cobol, pascal etc.

Come to think of it, I find it much better, rather natural, to think of a business domain in terms of units of data, its properties and its mutators. I’ve also yet to see a truly database-agnostic crud-based business application that’s not built around either one of a service oriented or message driven architecture. I’ve also seen less than a handful applications (among hundreds) that actually changed database vendors.

Regarding data formats, neither of those packages that I’ve set out to use actually describe how data is persisted. They are merely javascript object literals that either validate (set), mutate, or get by the app container’s functions which are pure javascript. It so happens that mongodb api is fully javascript compliant.

In fact, the closest that gets to an actual OO abstraction in meteor/mongo world is https://github.com/jagi/meteor-astronomy which keeps everything defined within a single Astro.Class and replaces all the packages I’ve referenced in my question. But hey, I’m not feeling comfortable enough with Astronomy to start off a project with it.

Jump here if you feel like reading, but skip ahead if you want the TL:DR

So yes, I believe that the properties defined in simple-schema and collection-helpers do belong in the same file, if not in the same object literal because the simple-schema describes what’s on the db and collection-helpers contain/describe:

  • some calculated transformations
  • some getters
  • some custom type castings
  • relations to other database objects
  • some potentially denormalizable references

which in fact, in a traditional modern OO database abstraction/model layer, would have been kept within the same class only to be annotated (java) accordingly.

Jump here

I would have loved providing an actual example, but the app I’m working on is a very specific domain with names and properties that makes no sense to anyone who has not been given a proper introduction to, hence the trivial (and kind of silly) posts/comments example.

So, getting back to the original question, I’m looking for a way to do the following (or something similar)

**the posts “object”: **

// Stock meteor (client & server)
Posts = new Mongo.Collection('posts');

// Simple schema (client & server)
PostsSchema = new SimpleSchema({
  title:  {type: String},
  authorUserId: {type: String}
});

// Collection2 (client & server)
Posts.attachSchema(PostsSchema);

// Collection helpers (client & server)
Posts.helpers({
  // For example, this could very well be considered for denormalization some day
  latestComments: function() {
    return Comments.find({}, {sort: {commentedAt: -1}, limit: 3});
  }
});

**the comments “object”: **

Comments = new Mongo.Collection('comments');

CommentsSchema = new SimpleSchema({
  postId:  {type: String},
  commentedAt: {type: Date}
});

Comments.attachSchema(CommentsSchema);

Comments.helpers({
  postAuthorFirstName: function() {
    var userId = Posts.findOne(this.postId).authorUserId;
    return Meteor.users.findOne(userId).profile.firstName;
  }
});

So, when I decide to separate posts and comments into their own packages, posts requiring the comments export and vice versa become a circular dependency problem and I need an elegant way to solve that, while keeping the code readable in such a way that the properties of a post and the properties of a comment are self-evident.

PS: we are not discussing if making postAuthorFirstName or latestComments methods on their own would be better. For sake of argument, assume that they are inseparable properties. In fact, let’s assume that they are https://github.com/aldeed/meteor-collection2#autovalue autovalues that reside within the schema that need to be inserted within the same transaction, and not requiring a collection hook.

PPS: I’m not getting into publish composite, but what it does is (obviously on the server) publish the related cursors along with the requested primary cursor. In our case, it would run the same find queries from the helpers to fetch and publish a cursor containing the top 3 comments along with a post as well as the post and author profile of a comment.


#5

thanks @entropy that’s probably the way I would go were it actually a posts/comments problem. In my case, posts/comments are just hypothetical placeholders for any two interdependent objects.


#6

Serkan i think you have many more experience and knowledge than me, so what i will say will probably not be a surprise, but it seems you are trying to square the circle. When you have circular dependencies, it probably means you have a coupling problem. I don’t know how you can break your dependencies problem without decoupling.
Either you think there is strong cohesion between the different dependencies and then you should make them a unique package, or you think the different parts should be separated and then if you can’t just put them in different packages without introducing circular dependencies that means they are still not decoupled enough !
That’s the problem with “simplicity”, there is a time when it’s too simple to be manageable.


#7

Thanks @vjau I doubt I have more knowledge, even if I do, I’ve buried myself too much with a simple problem, that I guess I am not thinking clearly :smile:

My problem is, I’m still trying to wrap my head around the execution order of javascript functions as they are scattered around packages/modules. It’s fine when they reside in the same package, though. That’s what’s actually puzzling me and leading me to question what I do and don’t know about the packaging system.

Seriously, though, thank you for being patient with me :smile:


#8

Just put all your collections, schemas and collection helpers into one core package :wink:

File loading order:

  • Schemas
  • Collections
  • Collection Helpers

Folder structure example:

  • posts/
  • posts_schema.js
  • posts_collection.js
  • posts_collection_helpers.js
  • comments/
  • comments_schema.js
  • comments_collection.js
  • comments_collection_helpers.js

Rebuild performance causes on a complex meteor app
#9

Haha, thanks @Sanjo I guess this should have been very obvious since it
had exactly been what I’d used to do, create a models library (meteor
package), in my java projects, to contain packages (meteor folders) of
individual persisted objects.

I guess I was too preoccupied with trying to split things by high level
functionality that this never occured to me :slight_smile:

Thank you.


#10

UPDATE:

I’ve decided to start off with jagi:astronomy which combine features from all those collection related packages with a nice API.


#11

Let us know how it goes, but remember, “ORM is the vietnam of computer science”. :wink:


#12

Haha, if I forget about that, it will remind me of itself :smile:

My primary concerns are:

  • validation of the data on both client and server
  • create reactive joins
  • create transient properties
  • do these where I can read the code within the same context, preferably within the same file (but also be able to separate just in case)

I’m into my first few hours with astronomy now and have not yet regretted it. Development pace of the package is quite good. It is very easily extensible. Looks like a breeze to define new modules/behaviors/events not having to depend on multiple packages from multiple authors.

It does have access to both the underlying collection (javascript) objects (data) and the astro.class instance which is more of a traditional object representation of the model.

So @vjau 'll try to update as I go along, but by the looks of it, I’ll be reporting more on its github tracker since the author @jagi seems very responsive and open. :clap:


#13

@serkandurusoy: Can you give a quick update on this one? I just stumbled upon the same issue.

Did you have good experience with astronomy?
Did it help at all with the cicular dependencies within a package-based project layout?

Right now I have those packages (sticking to your example)

1) Posts (namespace App.Posts.*)
2) Comments (namespace App.Comments.*)
3) Core (api.use(['Posts', 'Comments']));

My main problem right now is that (due to load order in package based structure) I can’t access App.Comments.* within App.Posts.*.

But also I am very interested on your take on astronomy vs. this stack:

PLUS:
Would astronomie work with aldeed:tabular, or have a similar solution?

:smile:


#14

@thebarty astronomy vs collection2 does not make that much difference in that regard. I played with astronomy on two small projects, but have not yet started anything big. It seems mdg is also kind of endorsing simple-schema now so I’m actually now more inclined towards SS until the dust settles. I’m sorry though because astronomy is really clean as an orm layer.

So, to answer your initial question, I put all of my collection definitons in the same package, while namespacing them. Then all other packages depend on that collections package. Also, I guard against undefined queries, therefore I have not faced circular dependency problems so far.

Astronomy works with tabular, no problem there.


#15

@serkandurusoy: Thanks for your reply! Yeah astronomy looks awesome!

OK, but is the solution to pack all collections, schemas and stuff into ONE “lib” package not totally destroying the idea of a package-based project structure? :smile:

I mean: I originally switched to this structure in order to clearly seperate my use-cases and still have them being able to “talk” to each other… and this circular dependency stuff is messing it up.

Right now I have this structure (=load order):

// PACKAGE ORDER (LOAD-ORDER from top to down)

// 1) LIB package (3rd Party and common stuff)
lib-package

// 2) usecase-* packages (1 for each usecase. each "usecase-*" package loads from "lib" )
usecase-comments // (want's to access posts)
usecase-posts // (want's to access comments)

// 3) app (loads ALL packages from 1) & 2) and combines them as an App
my-app-package

#16

Well, to be honest, I initially tried to separate packages as “self contained modules” but then decided that the app would not be extended by third party “modules” and a clear picture of the complete app data model sitting in one place (package) sounded like a much better idea (that’s what I’m used to from java packaging as well).

So I created a few base packages for namespaces, common meteor packages, common third party packages, a data model package, a utility package, a few utilities that deserve their own packages, and then separate packages for each feature. The feature packages mostly contain template code and logic while some of them having their server counterparts as separate packages as well. And that’s mainly because, for that app, I needed different admin and user interfaces so reused some of the packages, while separating others into their respective apps.