Materialize Breaks Spiderable


#1

I am trying to use spiderable on Modulus but nothing seems to work. I have tried the following packages without any success.

dfischer:phantomjs + ongoworks:spiderable:

spiderable: phantomjs failed: Error: Command failed: 
    at ChildProcess.exithandler (child_process.js:658:15)
    at ChildProcess.emit (events.js:98:17)
    at maybeClose (child_process.js:766:16)
    at Process.ChildProcess._handle.onexit (child_process.js:833:5) 
stderr:

dfischer:phantomjs + spiderable:

spiderable: phantomjs failed: Error: Command failed: Can't open '/dev/stdin'
    at ChildProcess.exithandler (child_process.js:658:15)
    at ChildProcess.emit (events.js:98:17)
    at maybeClose (child_process.js:766:16)
    at Process.ChildProcess._handle.onexit (child_process.js:833:5) 
stderr: Can't open '/dev/stdin'

#2

Are you using SSL? If so you might try gentlenode:spiderable-ssl


#3

I know that this is not exactly what you wanted to read but : don’t use spiderable. Specially on Modulus. It’s painful to configure properly and it can be harmful for your server. If you have more than a few pages pages (let’s say more than 500 ; if you have blog posts, comments or things like that, you are quickly going to that kind of numbers), search engines, social networks and others are going to crawl you site several times per minutes. Spiderable take 5 to 15 seconds to render your page (mostly waiting for connections and datas). You don’t want that to happen on your server, And you want to cache the result of the page generation.

Look at what those guys do : https://prerender.io/.

Four bigs advantages :

  • it’s a 5 minutes setup
  • crawlers will never make your server work
  • once the page has been rendered by their server, they cache it (in an s3 bucket I think). So the page will be serve in 5 to 50 milliseconds to the crawlers. They will like it a lot
  • it’s free up to 250 pages

I use the statup plan : up to 20.000 pages with a 7 days cache freshness, it’s 15$ / month. It will take three years to cost more than the three hours you will loose configuring spiderable on Modulus. (depending on how much you charge per hour).


#4

@fabienhuet Prerender looks great, did you have any issues setting it up with meteor?


#5

@Everhusk When I say this is a 5 minutes setup, I’m not exaggerating.

1 - you create an account at https://prerender.io/account (the longest part)
2 - you copy your token here : https://prerender.io/install-token
3 - you add the corresponding package from atmosphere : meteor add dfischer:prerenderio
4 - you add your token server side (I choose the js way over the setting way : prerenderio.set('prerenderToken', 'YOUR_TOKEN'));

And, that’s it.

( For me, this is all temporary. I will go full SSR with flow router and react as soon as the solution is production ready. But in my opinion, prerender is the best solution at the moment.)


#6

Sweet, thanks! I was looking for a fill in solution as well :slight_smile:


#7

I’m not even using SSL yet…


#8

Thanks for the suggestion @fabienhuet. I actually tried using prerender.io and dfischer:prerenderio but the package wasn’t working for me. I get the same issue as https://github.com/dfischer/meteor-prerenderio/issues/7.


#9

I found out that materialize was causing the problem. I’ve submitted the issue https://github.com/Dogfalo/materialize/issues/1805. Any help would be greatly appreciated!