Meteor App MongoDB cursor not returning correct number of data

cagapejuvy · November 15, 2017, 12:29am

Hello Guys,

Just want to share my problem regarding the issue I’ve encountered just in case there’s some geeks people who can help me.

My problem is that… we are developing a meteor app using mongo as database and we have large number of data. Sometimes our query results gave us only few data or less than number of data of what we expect. We already used two methods of fetching the data:

Jobs_status_identifier.find(_where, _clr).fetch();
Jobs_status_identifier.find(_where, _clr).forEach(function(_d){});

…but still it only gave us 35 records where in facts we have 41 records for that statement in the database.

The mongo counter:

Jobs_status_identifier.find(_where, _clr).count();

it says… 41 but the .fetch and the .forEach only shows 35 records.

Does anyone here experienced the same or related problem?

TY

townmulti · November 15, 2017, 2:14am

The _where and _clr would make the difference.

cagapejuvy · November 15, 2017, 6:42am

thanks for the reply, nope that’s not the cause. As you can read my explanation on the part of .count() they have the same parameters but not the same output of numbers of data when I tried to .fetch().length.

robfallows · November 15, 2017, 8:19am

Need to see more code, as @townmulti says. Also whether this is on client or server - and any other differences - especially pub/sub queries.

cagapejuvy · November 19, 2017, 10:17pm

here’s the full codes, sorry it’s on the server side.

Meteor.methods({
_get_per_colour_jobs : function(_where, _clr){
_get_lists = Jobs_status_identifier.find(_where, _clr).fetch();
return _get_lists;
}
});

robfallows · November 20, 2017, 10:27am

I think our definitions of “more code” don’t quite line up

I actually do want to see what’s in _where and _clr. I also want to see a sample of the MongoDB data.

So, if you find().count() on the server, you get a different result from doing find().fetch().length on the server?

cagapejuvy · November 20, 2017, 10:37pm

hi, seems like the confusing things here is how I executed to identity the number of results. The same on the server side, this is how I execute the codes to find the difference.

in my method:

_get_per_colour_jobs : function(_where, _clr){
        console.log(_where);
        console.log(_clr);
        console.log(Jobs_status_identifier.find(_where, _clr).count());
        _get_lists = Jobs_status_identifier.find(_where, _clr).fetch();
        console.log(_get_lists.length);
        return _get_lists;
    },

Regarding what are the value of _where and _clr, here is it.

_where = { 'complete_details.address.state': { '$in': [ 'NSW', 'QLD', 'SA', 'TAS', 'VIC', 'WA', 'NT', 'ACT' ] },
'complete_details.claim.type': { '$in': [ 'Promise Claim', 'Other', 'Maintenance', 'Insurance' ] },
current_status: 'In Progress',
in_progress: { '$gte': 100 } }

then

_clr = {}

and the results in console log are:

I20171120-22:35:18.837(0)? 54
I20171120-22:35:19.840(0)? 45

as you see, that’s a new number of data but still the results are different with the same query.

full logs:

I20171120-22:40:42.682(0)? { 'complete_details.address.state': { '$in': [ 'NSW', 'QLD', 'SA', 'TAS', 'VIC', 'WA', 'NT', 'ACT' ] },
I20171120-22:40:42.683(0)?   'complete_details.claim.type': { '$in': [ 'Promise Claim', 'Other', 'Maintenance', 'Insurance' ] },
I20171120-22:40:42.684(0)?   current_status: 'In Progress',
I20171120-22:40:42.684(0)?   in_progress: { '$gte': 100 } }
I20171120-22:40:42.685(0)? {}
I20171120-22:40:42.924(0)? 54
I20171120-22:40:44.433(0)? 45

robfallows · November 21, 2017, 12:38pm

There’s something wrong with how you’re providing _where. If you console.log a JavaScript object, you don’t get multiple lines. I think you may be using a string instead of an object.

cagapejuvy · November 23, 2017, 11:20pm

That console log value of _where parameter is working perfectly on Studio 3T that we are using to manage the DB. It returns same as the .count() method result in my codes. So the issue is on the .fetch() method of meteor, sometimes it can read the data correctly and sometimes not. We have more than 20K documents in a collection we tried to test or under this collection Jobs_status_identifier. We still encounter this issue until now in some other parts of our system.

robfallows · November 24, 2017, 10:18am

I’m sure it is. If you copy and paste into a query tool, including the MongoDB shell, or Studio 3T, the conversion from string to object is managed for you.

However, if for example, you use a form to collect fields, which you then assemble into a query, that will result in a string which looks like an object, but isn’t. I’m not saying you’re doing that - it’s an example of how this might occur.

Change the top of your method to this:

_get_per_colour_jobs : function(_where, _clr){
  console.log(typeof _where);
  ...

and report back.

Less likely alternative answers are:

MongoDB is broken.
This happens to everybody, but no-one’s noticed.
You’ve dropped the unique constraint or entire index on _id.

cagapejuvy · November 27, 2017, 6:01am

hi @robfallows , … the console log results is ‘object’. I put it back to pub/sub method since the result is the same. Now, here’s the _where value I have on subcription.

{"current_status":"In Progress","in_progress":{"$gte":100},"complete_details.address.state":{"$in":["NSW","QLD","SA","TAS","VIC","WA","NT","ACT"]},"complete_details.claim.type":{"$in":["Promise Claim","Other","Maintenance","Insurance"]}}

The data in the DB is 53 but I got only 44 results when executing .fetch() function.

Additional details, on how I execute it.

_subsc_Jobs_status_identifier = Meteor.subscribe('Jobs_status_identifier', _where, {sort : {_number: 1}}, function(){
            let _get_lists = Jobs_status_identifier.find(_where, {sort : {_number: 1}}).fetch();
            Session.set('job_get_lists', _get_lists);
        });

Thanks

robfallows · November 27, 2017, 3:27pm

A couple of things to try:

Look at the data from find and fetch - what’s missing? Can you see a pattern, or anything odd?
Go back to methods and (in the method) use the rawCollection functions, which bypass minimongo. Note that the underlying MongoDB library has toArray instead of fetch:
```
Meteor.methods({
  async _get_per_colour_jobs(_where, _clr) {
  _get_lists = await Jobs_status_identifier.rawCollection().find(_where).toArray();
  return _get_lists;
  }
});
```
Note also, the use of async and await. I’ve also removed _clr, as it’s empty and not a valid parameter in the MongoDB find (Meteor’s find options are done differently). If you want to research this, check the documentation for find and cursor methods.

storyteller · November 27, 2017, 8:32pm

I think this has already been confirmed as a bug:

robfallows · November 27, 2017, 8:37pm

Except that neither skip nor limit are used in @cagapejuvy’s query.

cagapejuvy · November 28, 2017, 6:49am

hi @robfallows, as I’ve explained in previous reply I’ve moved it back to pub/sub method of getting the data from database. Maybe it can elaborate this issue by posting my publication (see below).

Meteor.publish("Jobs_status_identifier", function (params, _clr) {
    if(params){
        _get_resp = Jobs_status_identifier.find(params, ((_clr) ? _clr : {}));
        return _get_resp;
    }
});

Was just using the server side to test if the query is working or not and since it’s working the same as my pub/sub method then I decided to put it back as before for some reason, we need it to be reactive.

You mention that _clr is not valid to place in the .find()? The _clr here is flexible sometimes I put order by and limit params and sometimes not, that’s why I put a condition to convert it to {} if not defined. Can you explain why should I have to remove it there?

Anyway, I still use the same construction of _where to this pub/sub method and unfortunately the issue still exist, not in all queries but in some of my queries like I’ve posted above previously.

Thanks!

cagapejuvy · November 28, 2017, 6:51am

@robfallows, maybe is convered in the issue posted by @storyteller ? Hopefully it can be fixed soon.

robfallows · November 28, 2017, 10:26am

Only if you want to test with rawCollection().

None of your examples or information have said this. In fact, you have explicitly stated that _clr = {}.

If you are not prepared to help by stating exactly what you’ve done, then I’m not prepared to spend any more time on this.

cagapejuvy · November 28, 2017, 11:22am

hi @robfallows, sorry if I missed something you requested. That server method is not being used already like what i’ve said because we want it reactive. Here’s what I did on how I execute the query.

first I constructed this _where params, on server side

_where = { ‘complete_details.address.state’: { ‘$in’: [ ‘NSW’, ‘QLD’, ‘SA’, ‘TAS’, ‘VIC’, ‘WA’, ‘NT’, ‘ACT’ ] }, ‘complete_details.claim.type’: { ‘$in’: [ ‘Promise Claim’, ‘Other’, ‘Maintenance’, ‘Insurance’ ] }, current_status: ‘In Progress’, in_progress: { ‘$gte’: 100 } }

then I run these two, this time on server side @ main.js

console.log(Jobs_status_identifier.find(_where).count());

_get_lists = Jobs_status_identifier.find(_where).fetch(); console.log(_get_lists.length);

and the result still different. Don’t be confused with _clr I already remove that.

cagapejuvy · November 29, 2017, 3:21am

Another interesting query that give me only more than half of the real results. The total data is 998 but the .fetch() method only give me 558. This is funny, I created only a simple query (on server side) which is below:

let params = {"$and":[{"$or":[{"waiting_for_appointment":{"$gte":100}},{"waiting_for_scope":{"$gte":100}},{"waiting_for_accepted_scope":{"$gte":100}},{"waiting_for_go_ahead":{"$gte":100}},{"waiting_for_confirmation":{"$gte":100}},{"in_progress":{"$gte":100}},{"ready_to_be_invoiced":{"$gte":100}},{"to_be_paid":{"$gte":100}}]}],"current_status":{"$in":["Waiting For Appointment","Waiting For Scope","Waiting for Accepted Scope","Waiting For Go Ahead","Waiting For Confirmation","In Progress","Ready to be Invoiced","To Be Paid"]},"complete_details.address.state":{"$in":["NSW","QLD","SA","TAS","VIC","WA","NT","ACT"]},"complete_details.claim.type":{"$in":["Promise Claim","Other","Maintenance","Insurance"]}};

And run/execute

_get_resp = Jobs_status_identifier.find(params);
console.log(_get_resp.count());
console.log(_get_resp.fetch().length);

And the console log results

I20171129-03:15:03.083(0)? 998
I20171129-03:15:06.377(0)? 558

Any thoughts on this?

vigorwebsolutions · November 29, 2017, 3:42am

If you run this on the server, what do you see?

console.log(Jobs_status_identifier.find({}).count());
console.log(Jobs_status_identifier.find({}).fetch().length);

Although, as @robfallows suggested, I think the actual differences between the datasets is the key to figuring this out, not examining the counts.