Oplog Observe Driver and $near, $geoWithin, $skip


#1

claims there is no oplog support for geospatial queries.

that was as of ~ Meteor .7

Is this still true? For queries using $near or $geoWithin on the server?

When will support for $skip happen? Currently I am relying on a simple hack: when request next page in a pagination scheme, change the query to only allow results $gt/$lt (depending on sort order) the previous page’s LAST item.


#2

Can’t say when support will appear, but it’s still not there:


#3

@robfallows thanks for the information!

but it is quite unfortunate. i am sure they will do it soon - geolocation is becoming pretty important in the current wave of apps.


#4

I worked on Oplog tailing for $limit and as far as I remember, I couldn’t come up with any efficient way to implement $skip w/o storing half of the data set in memory. (which is obviously not something you would like to do, ever)

It is important to note that operations like $skip is actually very inefficient in the MongoDB itself (regular queries), so using it in any queries is not advised (static, or live queries).

Also, in RethinkDB, the change-feeds feature also prohibits the use of skip as it is super inefficient in all cases mentioned above.


Best way to do a large publication?
#5

This is exactly how I would recommend doing it. One of the core engineers of RethinkDB told me this is what they recommend to their users too.


#6

@slava thanks for the reply!

Right, but the small problem with the so-called ranged query implementation is the case where the sort field contains duplicated elements. In the case of real numbers, highly unlikely. But unless one could guarantee uniqueness, then something like $lte/$gte would have to be used and the user would be forced to see the last element of page N-1 as the first element of page N.

Not a huge deal, but still a small caveat. Any updates you can give us for $geoQueries? The comments in the snippets above are a tad confusing - what are the main problems with $geoWithin and $near?


#7

I think even if you have duplicate values, you can uniquely order elements by a compound comparator such as “compare by date, then if equal, compare by _id”? As far as I understand MongoDB supports compound indexes too.

For geospatial queries, it is just difficult to implement the exact logic of geospatial selectors in Minimongo, that is used in oplog tailing to incrementally recompute the query. Since there is work with floating point numbers involved, it is harder to replicate the behavior of MongoDB’s C++ code in JS. Also, the code comments explain some weird behavior of $near.


#8

Ah I see well that’s good to know. I do recall that about sorting in mongo, didn’t think it would do it by _id…

I just hope the geo queries have not been taken off the to-do list!


#9

Just had my share of frustration with this. Limiting and sorting is pretty awkward with $near or $geoWithin. This way is nice and simple :slight_smile:

Posts.find({
  start: { $gt: time },
  lng: { $gt: lng-r, $lt: lng+r },
  lat: { $gt: lat-r, $lt: lat+r }
}, {
  limit: limit, 
  sort: { start: 1, title: 1 }
})

#10

this is an elegant solution. I wouldn’t mind that the region picked out is
approximately square.

one thing though is that this still can’t sort by distance to the point in
question. that would have to be done external to the query.

I’m curious, have you compared the performance of this query to the
equivalent geowithin version?


#11

Well I wasnt able to get geowithin to sort and limit. I was having all kinds of issues honestly. I havent checked the performance much yet – thats definitely something I need to do. I have created any indexes either…

Cant sort by distance. Thats correct. Luckily I didnt need to though…


#12

geowithin sort and limit can be done normally in the specifier, but if you
want to sort by distance, use $near.