I am working on a revamp of the spiderable package, so i wanted to ask the community for a quick feedback about the question in the title.
Currently i am working on:
Replace phantomjs with zombie
Planned are also new settings which can be defined in Meteor.Settings:
- host (name or ip)
- allowed bot patterns
- caching, tmp lifetime
- precaching (reocurring process caches urls by time plan)