Making a web page crawler, available packages?


#1

Hello All.

I am looking for available Meteor packages that will help me crawl through a webpage and perform functions such as ‘login’ and navigate further into the site. Are there any such packages… I’ve come across a couple but they lack documentation and I’m not sure what available functions / methods exist.

thanks :smile:


#2

Did you try any of these:

https://atmospherejs.com/?q=crawl or https://atmospherejs.com/?q=scrape


#3

Hm, xray looks decent, actually! (https://atmospherejs.com/lai/meteor-xray)

I was wondering though why you, @rrrrrraul, are looking for Meteor packages for that when you should be (from my point of view; that is, I would be!) looking for npm packages for that and specifically how I can use phantomjs (from node) in order to do the crawling I need, and maybe look for packages that build on phantomjs etc, that make it easier for me to do what I need in terms of crawling and page interaction. :panda_face:


#4

I’m actually trying out anonyfox:scrape at the moment. Thanks for replying =)


#5

@seekr, interesting. I just didn’t know. MeteorJS is the closest i’ve got to node (i know it’s somewhere there under the hood). I’ll look into phantomjs.


#6

Hey, scrape maintainer here. I’d love to hear your experiences with scrape!

I’m refactoring it right now and any feedback could be incorporated.