Understanding Meteor as an more old-school developer

[65766988]

Hey everybody!

I’ve recently tried to pick up Meteor, but just as with many of the similar technologies I’m having trouble understanding the concept behind it.

First a bit of background info: I’m a developer, but not a web developer. However, I learnt the basics of the LAMP stack a few years ago, and found it to be very intuitive and, well, logical. I’ve since had a few smaller web development projects (Using LAMP), but nothing too big.

Now, one of the most often cited reasons I read for switching from LAMP to something like MEAN, Meteor, … is easier and faster prototyping; And I’d have to agree, coding something from scratch in LAMP (Or even when using a framework such as CodeIgniter or CakePHP) takes an absolute eternity.

However, there are some basics behind this, I guess old-school way of thinking, that I like - Namely, the structure behind everything.
You have a client requesting something from the server through a well-designed (or not) API, some logic runs on the server, updates the state of the database (or not), sends back a result, which could either be plain data or fully fleshed out HTML. Boom, simple.

Recently I have tried to start understanding all this new, more trendy stuff - Node, Mongo, React, Angular, etc etc.
I do get a very small part of it - For example, I enjoyed using Angular/React on the frontend to do all the templating etc, and then just request data from the server and replace it somewhere in the app.
But I have to admit: With everything else, ESPECIALLY the (js-in-the-)backend-part I’m utterly lost. I just… don’t get it. It feels… like it has no structure, somehow. You write some code, and suddenly that code is… on the server? the client? both? Then update some databases… But wait, it looks like the client just updated the database by itself (And also has access to the db in general)? What? And while I get the part where you’d want to put more of the work on the client than the server, how comes the client seems to suddenly be executing logic, instead of simple rest requests? And what even is Mongo, why I would I want to not have a structure in my database, and why would I not want the relational cross-referencing part?

As you can probably see, I am very confused. But reading articles or example projects online did not help a lot - Because especially most of the example projects are literally completely client-driven, which is exactly the part confusing me the most.

Would anyone mind trying to explain some of the above to me? I’d really appreciate it.

Thanks a lot!

[storyteller]

Hi @65766988,

I get what you mean, I also came from LAMP stack. What drove me to Meteor was that I could just build my app without having to spend days/weeks configuring all the build procedures compare to similar options at that time. But to your questions.

As even with the “old” stacks the structure you are describing are often best practices that crystallized over years of use (though I have seen some atrocious things where the MVC structure was completely disregarded and the code was all over the place, but it still worked). Meteor is still relatively young so in many places the structure is still being discovered. That being said the best practices are described in the Meteor Guide.

Front-end is self explanatory. Beside client you have two additional options for your code (for more details check the Meteor Guide section on file structure). First is for code that is both on the client and server. This is for code that you will need to access both on client and server, like your collections and schemas (aka structure for your database via things like simple-schema and Collection2 or Astronomy). Then there is server code (Meteor is build on top of Node.js which allows js to run on server). This code gets executed only on the server. This is where publications and methods are defined (in additions to other config code and things like server side rendering). What you put on the server (your server folder) can’t be placed into client.

Now for the confusion regarding client updating the database. There are few things happening here. If you have database update code written on the client then if collection allows for updates from the client then it can perform the changes. This though is highly discouraged for obvious security reason and what you should do is call a method, that you defined on the server, to which you pass the data and then the method will process them and update the database.
As for the access to the database, unless you are using the autopublish package, you limit the data that the client has access to via publications that the client subscribes to. The data that is granted to the client then gets stored in a mini version of the database on the client (minimongo) which you can search and retrieve data like from the main database (but only with the data that you have send to the client).
Meteor does not uses REST, it has something called DDP (Distributed Data Protocol), which uses the publications/subscriptions and methods to communicate between client and server via web sockets. This allows to push any relevant changes to the database to the client among other things. So instead of REST calls you are firing methods or subscribing to data over a direct connection to the server.

As for MongoDB (aka Mongo) it is not relational database that you are used to. It is a document store. If I over simplify it is a denormalized database stored as individual JSON objects. If you want to use relational databases it is possible via GraphQL, which you can implement in Meteor via say Apollo (which is from the creators of Meteor), but that is an additional level of complexity.

It is a lot of new things coming over from LAMP. I hope I was able to explain few things and point you to additional resources. I would be happy to answer any other questions you have (provided I know the answer), but let’s make it into bullet points so it is easily accessible for future readers as I’m sure there are others in the same situation like you (I was there once as well :D).

[coagmano]

Hey @65766988!
To add to the excellent advice storyteller has given, I found this article very useful to understanding where data lives and how it flows inside a Meteor app:

Data flow from the database to the UI: Three layers of Meteor

Although if you go down the more popular GraphQL / Grapher / Apollo route, the data flow is less Magic
as you have to explicitly fetch what you want and pass it around

[65766988]

@storyteller

Thank you very much for your answer! It did help a lot. I think I’m getting the basics a bit more now, although I’m also still missing a lot of stuff, but that will become apparent when I try to program something.

@coagmano

Many thanks to you as well! That article was indeed very helpful. Actually made me really interested in Meteor now.

I think I’m just going to try to start some mini-project, and post questions along the way if I can’t find the answer to them.

Edit: One thing I’m having a bit of trouble with are the subscriptions/publications. As far as I understood, subscribing to one part of a collection (I’m saying “one part” because it could be filtered) gives the client not only access to the state of that part of the collection at the time of the subscription, but also to future updates of that part of the collection. Am I wrong?
One thing I tried to do: A subscription that gives access to certain documents of my collection based on a Session variable (I might have used the session functionality wrong, though). The session variable then gets updated as certain things change in the client, and therefore I expected the local database to be updated as well as the server-side subscription changes. Is that not how it works - did I expect too much magic?

Here some code to illustrate what I mean:

//SERVER SIDE
Meteor.publish('suggestions', function(){
	var query = Session.get('searchQuery');
	//check(query, String);
	if(query == "") {
		//return default suggestions
	} else {
		return Dataset.find(/*{name: /query/}*/); // dont do this, but do some kind of tf-idf ranking and then limit by 20
	}
});
//CLIENT SIDE
Meteor.subscribe("suggestions");
Template.searchbar.events({
	'input .homepage_searchbar'(event, instance) {
		Session.set('searchQuery', event.target.value);
	}
});
Template.suggestions.helpers({
	'suggestions': function() {
		return Dataset.find().fetch();
	}
});

Wait, nevermind, one of the reasons it did not work without giving an error was that I had misspelled what I subscribed to. Now it complains about my use of Session, which I understand is client only (A bit different than old school webdev, haha). How else can I get a client variable to be used on the server side? I read about “Methods”, so I now tried this:

//SERVER SIDE
var currentQuery = "";
Meteor.methods({
	'search': function(query) {
		//check(query, String); + other db character escaping
		currentQuery = query;
	}
});

Meteor.publish('suggestions', function(){
	console.log('hello;' + currentQuery);
	//check(query, String);
	if(currentQuery == "") {
		//return default suggestions
	} else {
		return Dataset.find(); // dont do this, but do some kind of tf idf ranking and then limit by 20
	}
});
//CLIENT SIDE
Meteor.subscribe("suggestions");

Template.searchbar.events({
	'input .homepage_searchbar'(event, instance) {
		Meteor.call('search', event.target.value);
	}
});
Template.suggestions.helpers({
	'suggestions': function() {
		return Dataset.find().fetch();
	}
});

However, that does not work either. From that, I assume that subscriptions are not quite as magic, and instead only return the current state of the database, and not future updates? Any thoughts on the above?

Also, from what I got, files that are not in /client/ or /server/ will be included in both. However, when I made a collections.js file that was in neither folder, that simply created a collection Collection = new Mongo.Collection("MyCollection");, I could not access it anywhere - I had to declare it in the both of my main.js files?
And talking about main.js files, the link @storyteller linked about file structures talks a lot about imports and stuff, and that the main.js files should not be used for any logic itself - Is it that bad practice to put the logic in there? (I did it because the default app structure generated by the CLI had logic stuff in there, and I’m still very unfamiliar with the whole js class and import system.)

Anyway, thank you all very much for your help!

[jorgeer]

I see that you are a bit off in the mental model of the server and client, so I’ll see if I can help you out.

Server side
The code you have on the server will have to work for all the clients it serves, so when when you declare variables like var currentQuery = "";, you are assuming that you are only talking to one client at a time, and storing state about that user in the server’s “space”, which won’t work. The equivalent in a “normal” framework would be to save a global variable in a controller after getting a GET request, then using that variable in another controller from another GET request, it’s just not supposed to work like that. You have to provide the search in the payload of the request.

Methods
You will declare a set of methods, which can be called from the client. They work like simple endpoints, like a unified GET/POST, which can be used to send data, or get data using the callback argument. All communication with methods are initiated by the client. So you do a “Meteor.call” function call to reach a method definition on the server. If you declared a method called “search”, you reach it with Meteor.call(("search", (error, result) => { ... do something with result }).

In the method definition, you can do database lookups and whatever you want, and return some data if you want to retrieve some data, but you should probably not be altering some global variables here. You should be storing your state in the database.

Subscriptions
I like to think of this as “Synchronize the result of the query in the publish function with my client-side database(minimongo), so that I can then query for it client side as if I had access to the actual database, and keep that data updated if it changes in the actual database.”
This results in you having a live subset of the actual database at your fingertips on the client, which you can feed into your react components to render. This is immensely powerful if you have multiple users looking at and changing the same data, or just one user changing his own data and seeing it update in (soft) real time.

Publish
Here you are declaring which data you will allow the user to have access to, if he subscribes to that function. You can provide arguments to the publish function to get exactly the data you want, so if you for example want to see all “suggestions” matching a search, you’d have something like this:

// Server
Meteor.publish({
  suggestions (search) {
    return Suggestions.find({ title: search });
  }
})

// Client

// Tracker.autorun will run the given function again if a "reactive source" like Session.get changes its value.
Tracker.autorun(() => {
  const searchValue = Session.get("search");

  // Tell the server to send you the data from the "suggestions" publish function, and Meteor will stuff
  // that into its client side database. You can then query for that data elsewhere.
  Meteor.subscribe("suggestions", searchValue);
})

You can now query for the suggestions in your template.

[65766988]

@jorgeer

Thank you very much for your explanations!

Yeah, I indeed misunderstood how the server side works. I was still thinking the more traditional way with “one execution per client”.

I’m still a little confused with Methods and Subscriptions, though. Like, I get the basics I think, but not so sure about some details.
To make the question more concrete, let’s take the example of what I wanted to achieve above, and what you corrected in your post.
Initially, I thought that using subscriptions here instead of methods would be nice, because I could kind of do the “search query updating” and “suggestion fetching” independently, and everything would basically happen automatically. However, since you showed me that a tracker (Which is quite interesting, by the way) is needed and the client has to subscribe again at every change in order to pass the argument - So what would be the advantage of using the subscription here instead of a method? I guess I’m a little confused on when to use methods vs subscriptions. Or, generally, what to use either for.
Then for templates/spacebars, I’m not quite sure I get what I have access to with helpers and what not. Say I return an Object in my helper, but my helper does take an argument. Do I go something like {{helperName argument.propertyName}}? That seems slightly weird (and doesn’t work, anyway), but there must be a way to access the properties of an object returned by a parameterized helper, right?

Also, I’m not sure if I understood the use of the Tracker correctly. Does the tracker update on ALL detected reactive updates? Or just on, magically, the one used in that specific case - Here, the Session?
And why does the searchValue need to be a constant? I would understand if it could somehow be concurrently modified, but that isn’t the case, right? Since each run of the tracker will instantiate a new searchValue?

Ah, and finally, if I subscribe to something often, do the “new results” get merged in the local minimongo with the old ones? Say, I subscribed to a channel “hello world”, that alternatively started returning document 1 or document 2 from a database. After 2 subscriptions, would I have both documents in my minimongo or only the last one? If the former is the case, do I need to empty the whole minimongo manually before subscribing again? Edit: Nevermind, it luckily doesn’t merge the results. Great.

Thanks for the help :)!

[jorgeer]

So there was a lot to unpack here! Trying to swallow the whole framework with all its strange components is hard, hehe, especially when we begin talking about the client frameworks as well as the whole conceptual model of Meteor.

“one execution per client”.

Meteor methods really are one execution per client. However, I think you are confusing the initial definitions in the server file with the execution of a request. The statements in the server file(s) only execute once at startup, so when we interact with the server later it is though the functions defined as methods and publications, just like you’d define controllers and routes in a regular MVC framework. Within those confines, Meteor handles requests one by one.

a tracker (Which is quite interesting, by the way) is needed and the client has to subscribe again at every change in order to pass the argument

If you only execute the subscription once, you will only synchronize results that match the initial search. So tracker will intelligently (magically) take note of things that change within that function, and help us by subscribing to the new search value. The advantage would be that you have a copy of all the matching search result in your local copy of the database, and you can do what you want with that, query in the data in multiple places for display. If the data you have searched for changes while you are looking at it, it will also automatically be updated. With the autorun, you don’t have to manually call the subscribe function again “yourself”. If you change the session value, autorun will run the function again, and you will now subscribe to a different query that matches your search result.

I hope I am able to convey that the autorun thing is only necessary if you change the argument to the subscription.

So, in general use subscriptions when:

• You want the data to be synchronized with changes in the database
• You want to use the data in many places, and be able to query the local database for the data

Use methods when:

• You don’t care about the data always being up to date
• You want very fast data-fetching
• You want to send data to the server

In this case, personally, I would probably go with a method. Usually you wouldn’t expect / care that some suggestions change or are updated as you are searching for them. You only want exactly the results for the search you are typing.

Subscriptions are for cases where the data changes, and you want to receive those updates automatically, without having to resort to polling the database, as you would in other frameworks. Say a chat room, you want all the latest messages in the room to automatically be synchronized with your client and not ask the server every 10 sec whether there are any new messages.

Regarding the Blaze template questions, you’ll need to provide some more code for it to make sense I think.

[jorgeer]

@65766988

And why does the searchValue need to be a constant?

You might not be familiar with the latest and greatest syntax of javascript, but let and const are new ways of declaring variables. Good blog post about it

It doesn’t need to be constant, but I find it good practice to make all variables that shouldn’t change constant, in this wild dynamic language of ours. The last few years, I can probably count on one hand the number of times I’ve needed to use let / var, thanks mostly to functional concepts such as map, reduce, and libraries like underscore.

Ah, and finally, if I subscribe to something often, do the “new results” get merged in the local minimongo with the old ones

minimongo is not a simple object that gets overwritten with new results, it is a functional database that adds new “documents” if you subscribe to more data, or if documents are added to the server side database and synchronized.

I subscribed to a channel “hello world”, that alternatively started returning document 1 or document 2 from a database.

Both would be in minimongo, and you could query for them as if you were on the server. I don’t understand what you mean by returning doc 1 and doc 2. It doesn’t return things in that sense, it just makes sure that the results from the publish function are available on the client side continuously.

[SkyRooms]

Yeah, same here. Came from PHP + MySQL.

Meteor is so much better. It’s the future. I’ve been studying Meteor for 3 years and I’m FINALLY just getting good at it.

[coagmano]

Just a quick clarification on minimongo:

Minimongo does merge the documents from different subscriptions together
What’s happening in the example is that because the argument has changed, the original subscription stops, and the documents are removed from minimongo, and then the new documents are added.

If you subscribe to two different subscriptions that both return documents in the same collection, the results will be merged together in minimonogo

[jorgeer]

Yes, though from what I understood, he described a scenario where Doc 1 and doc 2 were merged into each other, resulting in 1 merged document, which does not happen. I might have misread that part of the question.

[cloudspider]

@jorgeer He means its just 1 document. Lets say 1 publication publishes a document from an articles collection. The article doc might look like this:

{
  _id: "12345abc",
  title: "Some Article",
  body: "Some awesome text"
}

Lets say you would publish this as part of a list of articles. You might not want the body yet, since that’s a large chunk you are not going to display in your list:

Meteor.publish('articles', function() {
  return ArticlesCollection.find({}, {fields: {body: -1}});
});

This would give you a list of articles without their bodies if you would subscribe like this in your blaze template:

Template.articlesIndex.onCreated(function() {
  this.subscribe('articles'); // Subscribe for as long as this template is active
});

This list of articles can be found in your clientside ArticlesCollection instance. ‘magically’. You are basicaly setting up a realtime connection between the server and the clientside collection where the publication keeps your collection up to date with the server.

Now lets say a user clicks on the article to view the complete version. You would have another publication for the details like this:

// No param checking. I know, not secure, but just for illustration purposes
Meteor.publish('articleDetails', function(id) {
  return ArticlesCollection.find(id);
});

In your details template you can now subscribe aswell:

Template.articleDetails.onCreated(function() {
  this.subscribe('articleDetails');
});

Now here’s the interesting part. If your articlesIndex template is still visible on the screen (not destroyed), the article information coming from the articleDetails publication will just be merged with the document already present on the client. More concreet, this happens:

// Published with the articles publication
{
  _id: "12345abc",
  title: "Some Article",
}

// Published with the articleDetails publication:
{
  _id: "12345abc",
  title: "Some Article",
  body: "Some awesome text"
}

This will result automatically in just one document on your clientside. This means that if both subscriptions are active, your articlesIndex template would now also have access to the body of this article.

The nice thing about this is that pub/sub is continously keeping your clientside up to date with data from the server, only filling it with new information instead of requesting it over and over.

[awatson1978]

I found it helps to think of minimongo as a client-side replica set. It will auto-sync with the main data warehouse depending on the pub/sub rules you define. Simple as that, really.