PhantomJS onLoadFinished or any other way to make sure page is loaded

[iffifan]

Hello

I am successfully using this library to run my scrapper.
I am facing 1 issue that some time scrapper misses some information due to that item is not available in dom. I am using Symfony DomCrawler and CssSelecter components.

I tried to add delay in request so that page is rendered properly before fetching content but it does not work properly because if i keep it low it misses information and if i keep it high it takes more time. So it is hard to find a proper number.

I just checked that PhantomJS has event "onLoadFinished"

Can we have something similar here?

Or any idea how we can achieve same thing?

Thanks

[thisispaul]

Hey,

I had to do something similar but with the onConsoleMessage event. I have amanaged to achieve this by overriding some templates (see here).

I have overridden http_default.proc and added in this just before the page.open:

/**
 * Wait for a specified console message
 */
page.onConsoleMessage = function(msg, lineNum, sourceId) {
    if(msg === 'phantom: ready') {
        procedure.execute('success');
    }
};

I have then overriden page_open.partial and removed all the code in there (i have just added a fallback for 10 seconds just in case the console message doesn't get fired for whatever reason)

window.setTimeout(function () {

    debug.push(new Date().toISOString().slice(0, -5) + ' [INFO] PhantomJS - Rendering page after delaying for ' + delay + ' second(s)');
    procedure.execute(status);

}, 10000);

You could do something similar but with the onLoadFinished event, as I don't think these things are currently built into the library.

[satinjeet]

@thisispaul the link does not work. do you have an example somewhere. i have been looking to override just the template with page.render to encapsulate under onConsoleMessage, but the services are not made to be customized. scratch that. i think i found it.

[satinjeet]

by the way. the correct link for custom scripts is Here

[dausruddin]

@satinjeet can you share to me how you make it possible?
I dont have any clue how to use onLoadFinished.

[thisispaul]

Ah good catch @satinjeet, don't know what happened there. Updated my post with it!

@psycholyzern, what is it you are having trouble with?

You should just be able to add the onLoadFinished event to the http_default.proc like I have done above but with the other event. Like this:

page.onLoadFinished = function(status) {
  procedure.execute(status);
};

Make sure you override the page_open.partial too as initial comment.

[dausruddin]

@thisispaul my setup just showing blank page on any website I tried to load. Idk why. Reinstalled a few times.

Edit: just a silly mistakes. Webpage are loaded now.

[Milkdog]

I'm not sure if this is the correct issue or not, but I have a similar case. Rather than just waiting for the page to load, I want to know that a certain element has loaded (Ex div ID #main-content). Then do the screenshot after that. The reason is that we have a loading screen before the actual content is displayed.

I have tried using PhantomJS's waitFor() function, but haven't been able to get it working correctly. You can see their example here - https://github.com/ariya/phantomjs/blob/master/examples/waitfor.js

If this is too different from the original request, I can create a new issue for it.

[jonnnnyw]

I have added a lazy loading option to the client which will wait for all resources on the page to load before rendering the page. This includes images, AJAX requests and any resources triggered by AJAX requests. This has been released in v4.5.

I will update the documentation tomorrow but basic usage is:

$client = $this->getClient();
$client->isLazy(); // Tells the client to wait for resources

$request  = $client->getMessageFactory()->createRequest();
$response = $client->getMessageFactory()->createResponse();

$request->setMethod(...);
$request->setUrl(...);
$request->setTimeout(5000); // Will render page if this timeout is hit and resources haven't finished loading

Basically the call to to $client->isLazy(); tells the request to wait for all resources to load and the $request->setTimeout(5000); sets a fallback which will render the page after a period of time. This is to safeguard against things like short period polling which will continually trigger resource loading.

You can lazy load all types of requests including screen captures and PDF output.

@Milkdog it might be best to open another issue for specific element loading and I can look into it for the next version.