Use asynchronous crawling

[umpirsky]

Currently, the crawler is running synchronously, meaning that it spends most of its time waiting for IO.
Using an asynchronous HTTP client would allow being much more efficient, by doing HTTP calls in parallel.

Note that the asynchronous behavior can be implemented in a fully BC way: $crawler->crawl() could still block until all URLs have been crawled, and the callback-based API can be kept the same.

This means that instead of using Goutte to send the calls, you would need to use Guzzle directly, to be able to access its asynchronous API. You could then convert the response back to a BrowserKit response before calling the callback if you want to keep BC (or you could decide to break BC and to pass a Guzzle response instead)

[umpirsky]

@stof I was thinking about this, and my initial idea was to use ReactPHP.

Your proposal sounds better to me and is definitely one of next things I want to do on this project.

Thanks!

[stof]

ReactPHP would not be a good fit here:

• its event loop is probably overkill here
• its HTTP client does not support HTTP fully (Gzip, etc...) because they are implementing HTTP themselves on top of their asynchronous socket client. Guzzle on the other hand relies on curl for the HTTP part, which is feature-complete.

[umpirsky]

@stof Thanks for sharing, I was aware of the 2nd point. 👍

[umpirsky]

@stof Here is the initial implementation. Feedback is welcome, thanks.

[stof]

@umpirsky turning an issue opened by me into a PR from your repo is really weird (and it makes it hard to understand things when looking at them later). It would be much better to open a separate PR referencing the issue

[stof]

this does not work, because links can be relative

[umpirsky]

@stof Yes, I know, sorry, I merged other branch into this so I created mess. You can focus on 346f2a2.