Thanks, @robfallows .
I am not sure what I am doing wrong but when I run APIKEY=“my-api-key” meteor --settings settings.json, console says ’APIKEY’ is not recognized as an internal or external command, operable program or batch file.
Do I need to do this?
Please suggest.
OK. You’re using Windows. On PowerShell you will need to do
$env:APIKEY="your-api-key"
meteor --settings settings.json
In cmd
set APIKEY="your-api-key"
meteor --settings settings.jsonThanks, @robfallows.
I got this running by setting process.env.APIKEY in ukgrid.js on server. Cheers!
Cool - just remember not to publish that code to a public repo, or your API key will be exposed!
Yeah… I understand that. Thanks.
Hi, Rob.
I need to query a collection and store all document objects into an array. I’ve been looking around for a solution on this without success. Is there any builtin way in Meteor for this?
An array in Javascript?
const myArray = MyCollection.find(...).fetch();
Where ... will be your query if you are selecting a subset of documents.
Thanks, Rob! I got around that a minute after I posted my question. Your answer confirmed that I found the right thing.
Hi, @robfallows.
I am back with new problem. 
Actually I am looking for suggestion how to achieve this. Here is what I need to do:
I am not sure how I will achieve #3 here. I know Meteor instantly updates client w/o page reload for changes but I am not able to figure out how a queue processing on server will update client. Please suggest what should be the way to do this?
So I googled around it and learnt that I should use Rocketchat:Streamer package.
I am trying to use this package but not able to get it working per my needs. As long as I keep streamer initialization and keep using streamer.emit() and streamer.on() from inside lib folder of my project, I can send and receive messages via browser console. As soon as I try to use .emit() from server folder and .on() in client folder, it just stops working. I need to send progress data from server to all logged in clients.
I may be doing some silly mistake to get that package work from server to client but I am not sure what that may be.
Please help!
Finally, I got it working. I am yet not sure what I was doing wrong earlier. I moved client and server code from lib file (leaving streamer initialization still in lib folder) in respective folders and it started working.
Thanks. Cheers!
Glad you found a solution. You could have used Meteor’s pub/sub to do this as well.
Thanks, Rob.
pub/sub means publish at server and subscribe at client. Right?
By what I’ve read about it so far, that works when you are working with collections. Server keeps inserting docs in collection and client keeps updating itself in real-time. Is there a way using pub/sub w/o using collections?
There is also one more thing that I need suggestions/help on.
I call the server method supplying an array. Server method needs to
When server is done doing this on all array items, it needs to send a flag back client.
I know Meteor.Async() helps doing things synchronously. should I apply Meteor.Async() on http.request calls? Here is my snippet:
.
.
.
for(var i = 0; i < selectedPages.length; i++){
var pageDetails = allPagesDetails.filter(obj => obj.pageUrl === selectedPages[i] );
/*var pageDirPath = url.parse(pageDetails[0].pageUrl).pathname;
var pagehost = url.parse(pageDetails[0].pageUrl).host;
console.log(pageDirPath);*/
// console.log(pageDetails);
if(i == selectedPages.length -1) lastPage = pageDetails[0].pageUrl;
getPageContent.getPageContent(pageDetails, function(err, response){
if(err){
console.log(err);
return
}
...
...
Fiber(function() {
try {
PageSitemaps.insert(response.pageDetails[0]);
sendMessage({
event: "processBuildProgress",
pageUrl: response.pageDetails[0].pageUrl,
statusCode: response.statusCode
});
} catch (error) {
console.log("Error while Saving to DB: "+error);
}
}).run();
}
}
//================ getPageContent ===============
function getPageContent(pageDetails, callback){
/*var pagehost = url.parse(pageDetails[0].pageUrl).host;
var pageDirPath = url.parse(pageDetails[0].pageUrl).pathname;*/
var options = {
host: url.parse(pageDetails[0].pageUrl).host,
path: url.parse(pageDetails[0].pageUrl).pathname
}
// console.log(options);
var request = http.request(options, function (res) {
var data = '';
res.on('data', function (chunk) {
data += chunk;
});
res.on('end', function () {
// console.log(res.statusCode);
// console.log(res.statusMessage);
var pageRequestResponse = {
statusCode: res.statusCode,
statusMessage: res.statusMessage,
content: data,
host: options.host,
pagePath: options.path,
pageDetails: pageDetails
}
for(i in res){
// console.log(i);
}
// callback(JSON.parse(data));
callback(null, pageRequestResponse);
// return data;
// savePage(data);
});
});
request.on('error', function (e) {
console.log(e.message);
callback(e.message, null);
});
request.end();
}
Yes …
You can consider a subscription as the client telling the server to do something. The publication can do whatever you want. Most times that will be a query against a MongoDB collection - but it doesn’t need to be.
The other thing that’s expected of pub/sub is: “while you’re doing that thing, keep me informed about progress while you do it.” Again, you’ll usually just use the inbuilt livedata in Meteor to ensure a collection’s changes are available to the client - but it doesn’t need to be.
… and no 
The way server-side changes are communicated to the client is via minimongo. However, it’s important to note that you don’t need an actual (disk-based) MongoDB collection for this. Instead the publication uses the pub/sub API to populate a client-side minimongo collection.
So, the publication could be written like this:
Meteor.publish('buildFiles', function(urls) {
// skipping the sanity checks and error checking for brevity
urls.forEach((url, index) => {
const data = HTTP.get(url);
buildHTML(data);
this.added('dummyCollection', `seq${index}`, { index, url });
this.ready();
});
});
Note that the above code processes the urls sequentially. You could use callbacks or Promises to process them concurrently.
As each url is processed, a new document is added (this.added) to the client-only collection, dummyCollection, and made available to the client (this.ready). The document imitates a MongoDB document, so needs an _id ('seq0', 'seq1', …). Otherwise, my example just has an index field and the url. You can, of course, add whatever you want.
On the client, you need to define a matching collection and subscribe in the normal way. Note that you don’t define this collection anywhere on the server - it doesn’t exist there.
const dummy = new Mongo.Collection('dummyCollection');
Template.buildFiles.onCreated(function buildFilesOnCreated() {
this.subscribe('buildFiles', ['url1.com', 'url2.com', 'url3.com']);
});
Template.buildFiles.helpers({
buildProgress() {
return dummy.find();
}
});
You can also make use of autorun and have reactive parameters in the subscribe, exactly as with any other subscription, for example for resubscribing with another url list.
Thanks a lot, @robfallows!
It took me sometime to understand your solution (novice here 
). I couldn’t re-write whole whole process as this but your snippet gave me insight to use HTTP.get() in-line so all pages are read sequentially. I did a small tweak and tested; it’s working so far.
I will update you once I am fully done. Thanks again.
Hey, @robfallows.
I am facing another challenge here. As I wrote earlier, I am using HTTP.get() to read URLs and their content. HTTP.get() call is inside a loop which runs over an array of URLs. As long as array length remains moderate say not more than 150 or so, my code runs fine and server keeps sending progress data to client vai Rocketchat:Streamer package.
But when array length is higher say around 1000, then my code stops arbitrarily and server stops sending data to client. It neither throws any error on server/client; it just hangs. If I bypass HTTP.get() execution, code runs completely. Here is the code snippet:
for(var i = 0; i < selectedPages.length; i++){
var pageDetails = allPagesDetails.filter(obj => obj.pageUrl === selectedPages[i] );
var pageDirPath = url.parse(pageDetails[0].pageUrl).pathname;
var pagehost = url.parse(pageDetails[0].pageUrl).host;
if(i === (selectedPages.length - 1)){
lastPage = true;
}
try {
const data = HTTP.get(pageDetails[0].pageUrl);
var statusCode = data.statusCode;
var pageContent = data.content;
/*var statusCode = '200';
var pageContent = '<h1>Dummy Content</h1>';*/
} catch (error) {
throw new Meteor.Error('Oops...', 'Some error happened while reading '+pageDetails[0].pageUrl);
}
var dirs = pageDirPath.split("/").map(function (ele) {
if(ele != ""){
return "/"+ele;
}
}).filter(arr => arr);
for(let j = 0; j < dirs.length; j++){
if(j > 0) dirs[j] = dirs[j-1]+dirs[j];
}
console.log(dirs);
dirs.map(ele => checkDirectorySync(targetDir + ele));
var pageDir = targetDir + pageDirPath;
console.log(`pageDir: ${pageDir}`);
checkDirectorySync(pageDir);
fs.writeFileSync(pageDir+"/index.html", pageContent);
// ==========================================
try {
PageSitemaps.insert(pageDetails[0]);
sendMessage({
event: "processBuildProgress",
pageUrl: pageDetails[0].pageUrl,
statusCode: statusCode,
buildInitiator: initiator[1],
buildInitiatorName: initiator[0],
selectedPagesNum: selectedPages.length,
isLastPage: lastPage,
currentPageNum: (i+1),
buildInitTime: buildInitTime,
pageProcessingTime: new Date(),
targetSite: targetSite
});
} catch (error) {
console.log("Error while Saving to DB: "+error);
}
// ==========================================
if(lastPage){
console.log("Build processing is DONE.");
Builds.update(currentBuild, {$set: {processingStatus: 0}});
return true;
}
}
This loop is part of a method on server which is supplied the URLs array from client.
Thanks in advance!
Without doing more in-depth analysis, this could be one of two things:
Meteor.call with Meteor.apply and set the noRetry option to prevent this.top and watch the %MEM column.I suspect #2 here.
I am on windows. Is process.memoryUsage().heapUsed appropriate to check process memory?