Hello,
I am using Meteor for a big application that must store a large amount of data and constantly show the user a small portion of the available data (and always the most up to date data). I use React for the front-end and regular pub/sub to sync the back-end and the front-end (I use the publish-composite package for some publications). Everything so far has been working well as the user inputted data by calling some method that inserted into the matching collection.
However, I now have to get some of the data in a different manner. I want the server to regularly (approximately every minute, sometimes more frequently and sometimes less, at most once per second) send an HTTP GET request to a REST API, parse the data, insert the relevant parts into the relevant collections, and I then need the client to immediately display the correct data.
The added complexities are that the HTTP request and parsing could take some time so must be done without blocking anything else (so asynchronously?) and that the client cannot really simulate the server call since it doesn’t have the required keys and passwords to access the REST API (it probably shouldn’t be sending HTTP requests anyway).
Currently I am trying to use Meteor methods to achieve this behavior (since that is what I’ve used so far). Precisely, one method (the update method) issues an asynchronous HTTP call and returns (the data is inserted in the callback) and another method calls the update method on all the objects that must be updated, then sets a timeout to call itself again after some time. The second method is initially called on startup. Furthermore, since I do not want the client to simulate, these methods are defined only on server side.
This method however is not working, when the entries are inserted, although they appear in the main database, they are not automatically pulled to the minimongo (i.e. the subscriptions do not seem to be updating). I am not sure why that is (have tried making the update method synchronous with Future and importing the methods to client side with a isClient check that returned before doing anything, but neither changed anything).
In general, I am wondering if using methods might not be an error. I do not really need latency compensation (I just need the data to display once inserted in server) and I do not need the client to run it. However I do need some way for the subscriptions to update when inserting (perhaps asynchronously) into the collections from server side. Is there a good way to do this?
Thank you and sorry for the long post!
TLDR: What is best way to repeatedly (every minute) asynchronously issue an HTTP request to a REST API from server, insert data into collection, and get pub/subs to update so that client is displaying correct data?