I tried searching this one but found nothing.
Has anyone attempted to build something which would let you show a progress bar to visualize a subscription loading?
My app only loads subscribes to a single document at a time, however it’s usually a pretty big document. Would be great if I could somehow show a progress bar for how much has loaded vs how much it will load.
I show a infinite loop animation from a template when I’m doing this.
This look like this :
Session.set('loadingSuscribe',true); // putting my animation template active
mySubscribe = Meteor.subscribe(),function(){
Session.set('loading_suscribe_calculConso',false); // removing my animation template
})
And here is simply what I do in html
{{#if loading_suscribe_calculConso}}
<div class="loading">
<img src="/icon/loading.gif"/>
</div>
{{/if}}
I don’t know if it can help you.
Yeah I have the same thing now, would much rather have a realistic loading bar though.
@msavin - I know with meteor toys you’ve got the DDP inspector, do you know if there is anything that might be used to make a progress bar for a subscription?? In my situation I only have one quite large document I wan’t a progress bar for… I am guessing it would be much easier to show a progress bar for multiple documents downloading but thats not what I have.
Hey man- unfortunately not possible. It would have been cool if each subscription gave back an array of all the IDs that it watches… there’s a discussion somewhere on GitHub… but unfortunately does not exist AFAIK.
mmmm, you could build your self some mechanism. First sub only for the doc IDs { fields: { _id: 1 }}, so you get the total number of the documents, resub with all the oher fields, then with observeChanges you could watch the added or changed (not sure which would do the trick) messages and increment a counter :-), which would give a you a realtime progress.
EDIT: Also to make sure the counter is correct, check for all of the fields when observe changes fires.
One option is to use a method call to get the data. You can show the loading state on your app until the method response comes back.
You could also display a dummy progress bar. Even if it’s not semantic, I think studies show that simply having one puts the end user at ease. In fact, I am pretty sure Apple uses a dummy progress bar when sending text messages.
You could publish a collection count, for example using publish-counts package
Then when you make the subscription you know the total amount of docs, and by observing the collection you can show a progress bar.
thanks!
yeah see most of these suggestions for a work around wouldn’t work because I am only subscribing to a single document. the document is just a large one.
i do have a loading screen with an animation, but i would have loved to be able to have a progress bar across the top of the page kind of like youtube does when you change pages.
so basically I would need a hook for the server to report how big a document is, and then something on the client to report back on how much of the document has downloaded.
Hmm…
//progress.js
let init = true, guessCount = 30;
let progress = new ReactiveVar(0);
Meteor.subscribe('posts-abc', /*optional: {limit: guessCount}*/);
Posts.find({/*query only what is published by posts-abc*/},{fields:{_id:1}}).observeChanges({
added: function(){
init && progress.set(Math.min(progress.curValue + 1/guessCount),1);
//use Math.min in case the number of docs was larger than our guess
}
})
progress.set(1); //in case the number of docs was smaller than our guess.
init = false;
//view.js
Tracker.autorun(()=>{
let percent = Math.round(progress.get()*100)
console.log("loaded "+percent+" percent")
})
The tracker is shorthand for any reactive function, like a blaze helper.
If the number of documents returned was smaller than our guess, you’ll see something like:
//console
"loaded 3 percent"
"loaded 7 percent"
"loaded 10 percent"
"loaded 13 percent"
"loaded 17 percent"
"loaded 100 percent"
That is satisfactory.
If the number of documents returned was larger than our guess, you’ll see something like:
//console
"loaded 3 percent"
"loaded 7 percent"
"loaded 10 percent"
"loaded 13 percent"
...
"loaded 100 percent"
//still not fully loaded. >:) we have lied to the user.
This is unacceptable.
While users like to see progress bars go from 5% to 100% suddenly (it’s oddly satisfying), no one likes to see progress bars that say 100% when the data isn’t even loaded. I know what you’re thinking, we’ll just cap it at 99% and then make it 100% when it is done loading. Please don’t do that, as it’s an even shittier UX.
The fix for this problem is to know how many documents you will be returning. One option is to somehow keep track of the length of arrays, whether it’s through publishing counts, or keeping a cache of array/collection/collection-slice lengths on your server, which you send initially as metadata to your user when they first connect. That’s pretty tedious, tbh.
“Don’t keep track of counts, just take the length of the array…
”
- Some React person.
Here’s one way to do it. Remember, you are going to be observing the cursor anyways, so let’s take advantage of that information, instead of publishing the counts separately. In the end, the relevant count is ONLY of the cursor we’re sending to the user. The entire Posts collection count is irrelevant if I’m sending Posts by Mary only.
//publish.js
Meteor.publish('posts-abc', function(args){
let query = getQueryFromArgs(args);
let cursor = Posts.find(query);
let count = cursor.count();
this.added('counts',"posts-abc", {count: count}) //only care about that initial load.
let handle = cursor.observeChanges({
added: (id, fields)=>{
this.added('posts', id, fields)
},
changed: (id, fields)=>{
this.changed('posts', id, fields)
},
removed: (id)=>{
this.removed('posts',id)
}
})
this.onStop(()=>{
handle.stop();
})
})
Effectively, the first document you send to the user will be information about how many documents you will be sending to the user – this is exactly the metadata we needed in our previous situation. Now, you can change the client like this:
//progress.js, improved
let init = true, actualCount;
let progress = new ReactiveVar(0);
Meteor.subscribe('posts');
Counts.find('posts-abc',{fields:{count:1}}).observeChanges({
added: function(id,fields){
actualCount = fields.count;
}
})
Posts.find({/*query only what is published*/},{fields:{_id:1}}).observeChanges({
added: function(){
init && progress.set(progress.curValue + 1/actualCount);
}
})
init = false;
Now, i haven’t tested any of this. It might be that some subtleties in the order-received of documents obtained over DDP ruins this solution. If your count is sent after your posts start getting sent, we have a problem. You can write some math on the client to fix it, though. Just keep track of the number of posts documents sent from posts-abc, and employ a guessCount until you have the actualCount, then fix the progress variable accordingly. Check for existence of actualCount every time you get a post. If it exists, use it, and the fixed progress bar. Else, fallback to the guessCount. I think you shouldn’t have to do this though, as you should get the count information before the posts information. Anyways, I warned you.
thanks @streemo, thats a really detailed post, and would work for most peoples scenarios.
but like I said, I am only subscribing to a single large document.
on a mobile device the document takes anywhere from 10ms to 1second to load on a fast network, I am just thinking ahead, people will eventually have much larger documents which will take longer to sync.
It would be awesome if in the subscription handle there were a flag for total data being subscribed to - and a handle for the client to check how much has come down the wire.
for example, something like below.
rather than have it document centric, have it by the bit, because not all mongo documents will be the same size so doing it by bits would make more sense. However there could easily be some document centric functions too…
let yourCollection = Meteor.subscribe('someCollection');
// If the collection isn't ready have some functions which return some useful information which could be watched.
if (!yourCollection.ready()) {
console.log(yourCollection.requestSize()); // Returns the current subscription request being loaded in bits.
console.log(yourCollection.requestLoaded()); // Returns the total number of bits already in minimongo
}
Then just make a method call before subscribing. Server always has access to the full collection.
So you can something like this, if your collection is Posts.
In a method requestSize = function() { return Posts.find().count(); }
Then when you subscribe on the client, you first make the method call to fetch the total count. After you have the count, subscribe and observe collection changes.
Meteor.call 'requestSize', (err, res) ->
if res
sub = Meteor.subscribe 'postsSub'
...
I don’t see why this wouldn’t work with very large collections.
Interesting, my apologizes for not addressing that. GridFS is a specification for storing files in MongoDB which are larger than 16mb. IIRC, The files can be smaller than 16mb if you desire. By default, it stores files in 255kb chunks. It allows you to query for the file’s metadata (size, etc.).
One thing you could do is:
//Fake it!!!
let rate = getAverageClientConnectionDownloadSpeedInKbPerSecond(clientId);
let fileMetadata = db.fs.files.find(documentId);
let fileSize = fileMetadata.length;
let noiseInMs = 1000; //because shit happens, ideally this will be set to 1 standard from the mean.
let expectedTimeToSendDocumentInMs = noiseInMs + ((fileSize/1000)/rate)*1000 //i.e. fileSize/rate
//1000 in denominator comes from bytes to kb conversion, 1000 in numerator comes from seconds to ms conversion.
//animate the progress from 0 to 100 over the entire interval expectedTimeToSendDocumentsInMs
let progress = new ReactiveVar(0);
let timeIncrement = 50;
let percentIncrement = timeIncrement/expectedTimeToSendDocumentsInMs;
let fakeProgress = Meteor.setInterval(function(){
progress.set(Math.min(progress.curValue + percentIncrement),1)
},timeIncrement)
//clean up your dirty, dirty charade
Meteor.setTimeout(function(){
clearInterval(fakeProgress)
}, expectedTimeToSendDocumentsInMs)
That’s one way to do it. The noise is there to be a safety for when the client’s download rate randomly deviates from the mean download rate.
This solution requires you to assume the client’s average download rate which can vary. If you can get a distribution of download rates over time (like while the client is derping around, send some bits in the background and get a distribution before they request the bigDocument 
) you can fix this by expecting that the client’s download rate will equal meanRate + oneSigma.
Another possible solution, which is a bit taxing on the server:
Split your document into many little subscriptions, and then just assume that each subscription contributes equally to the download time.
//server
let exampleOfbigDocument = {
_id:'blahId',
bigImage:'RawrBigImageDataHere8934rfnief039jf02j0i3fjd20...',
gargantuanImage: 'iAmBigger!!!!ksfjdhwufhjd',
listOfStarsInTheUniverse:['SiriusA','Sun','Alpha Centauri',...],
listOfLeptonsInTheUniverse:['electron1','electronInYourLeftArm',...]
}
//client
let imageHandle = Meteor.subscribe('onlyTheFirstImage', 'blahId');
let largerImageHandle = Meteor.subscribe('theGargantuanImage', 'blahId');
let starHandle = Meteor.subscribe('allTheStars', 'blahId');
let electronHandle = Meteor.subscribe('electronHandle', 'blahId');
let progress = new ReactiveVar(0);
Tracker.autorun(function(){
let c = 0;
imageHandle.ready() && c++;
largerImageHandle.ready() && c++;
starHandle.ready() && c++;
electronHandle.ready() && c++;
progress.set(c/4); //4 is the number of handles.
})
E.g. In the above example, when 3 handles are ready, we assume the progress is at 75%. When all four handles are ready, the entire document has been sent.
This is not very good since we’ll have many subs for a single document, but might be good enough and doable enough for your needs. If your document is huge, and each client only really needs only that document in their session, this might be OK.
For more on gridFS: https://docs.mongodb.com/manual/core/gridfs/
Check out: http://www.speedtest.net/.
You could try to implement something like speedtest to get an idea of the time scales (per client) that you can expect your subscription to complete in.
EDIT: Your problem could be solved with a lower level node api which provides callbacks on a per chunk basis. You might hack into that if you’re really interested in the progress bar. There might be a way to send the data via node’s lower level API, and then “trick” meteor’s DDP/server cache into thinking it was sent over DDP, so that when you subscribe via ddp, you can listen for changes without sending the document again.
I thought this also, but it looks like the limiting factor for him is the document size itself, so the progress needs to happen on a per document basis, rather than a per cursor basis. Another problem is that you have to query twice. Instead, the counting should be done inline with the original query via added/changed/removed in publish.
Yes you’re right, the counting has to be done on a per document base via added/changed/removed (observeChanges).
But to show progress (like 5 out of 100, or 20%) you need to know the total amount of documents.
This can only be done by making another db query.
But you’re assuming that more than one document needs to be fetched. What if he only needs one document? Then the count is trivially 1, and the progress is trivially either 0% or 100%, which isn’t useful. Publishing the counts of the query is a separate method for getting progress which breaks down when the result set size N is of order 1. (N ~ 1)
If N >> 1, the following should work, and it requires only one query, and you get the count as a side-effect of getting the data:
Meteor.publish('posts', function(){
//query once, not twice
let cursor = Posts.find();
let count = cursor.count();
this.added('counts','posts-count',{count: count})
return cursor;
})
Let me know if this doesn’t work. Also, I mentioned a few ways to “fake it”, if you don’t know the exact size of the result set, but can make an educated guess. The count is required for an exact progress, but it’s not required for some progress at all. It really depends on your data. If your result set, for some reason, is capped by an upper bound, or you are dealing with paginated data, then you can make fairly strong assumptions about the count variable without actually wasting time to get the value of the count variable. In the case with paginated data, you can guess the count perfectly without actually querying it:
Client: Give me 50 results for page 2.
Server: OK! Here’s 50 results, first let me see how many results you need!
Server: Oh wait, you wanted 50. Ok, no need to get the count.
Either way, @cstrat seems to be in a situation where N ~ 1, so our count solutions won’t be useful for him.