Repository navigation
Parallel scraping results in misses & duplicates #6
Description
What an awesome scraper platform! Got all geared up in no time.
However, as I found single page scraping to work just fine, parallel scraping with many (I had 79) URLs fails, resulting in missed URLs and duplicates while the total sum of fetched URLs is correct.
I suspect the reason to be the queuing implementation. I tried a little fix on scraper.js that produced results I was hoping.
Activity
Attempt to fix parallel scraping queuing issue (mape#6)
gaara87 commentedon Jun 27, 2011
I too faced this problem. its just 20 urls, rather than 79.
Is there a way to enforce a timeout?
deggis commentedon Jun 28, 2011
What I remember, I think timeout wouldn't help, I think I tried one arrangement. I'm not a JS guru, so I'm not deadsure what would be a rock solid fix for this, but mine worked for me at least :)
deggis commentedon Jun 28, 2011
(Whoops, Comment & Close was kinda too close.)
nickewansmith commentedon Jun 13, 2013
Try using cheerio instead of jsdom and then implementing your own queuing, worked for me!