rendering tests: Error: timed out after 20 seconds

[jfirebaugh]

Seeing a lot of build jobs failing now due to timeouts in the rendering tests.

* passed background-color default
* passed background-color literal
* passed background-color function
* passed background-opacity color
* passed background-opacity image
* passed background-pattern literal
* passed background-pattern zoomed
[Error: timed out after 20 seconds]

I don't remember seeing this before #2617 -- @kkaefer is it possible that the networking changes started triggering this?

[mikemorris]

Pretty sure I've seen this prior, but dismissed it as a random hiccup.

[kkaefer]

Weirdly, Travis reported the entire test run as passing despite individual failing builds. I tried to reproduce this locally, but couldn't. The node tests never stalled for me.

[jfirebaugh]

Something changed here recently. I don't recall ever seeing this, or if it happened it was uncommon. Now it happens very frequently.

[jfirebaugh]

@kkaefer Can you please take another look at this tomorrow?

[jfirebaugh]

Since still rendering hangs seem to be a recurring problem, we should probably introduce some diagnostic tools. I would probably start with a companion method to Map::isFullyLoaded that logs the status of the resource tree. I.e. a resource identifier and whether or not the resource is loaded, for each of the following:

style
 - sprite
 - glyphs
 - sources
  - tiles

[kkaefer]

I tried VMWare Fusion and VirtualBox and was unable to reproduce this issue anywhere.

[jfirebaugh]

https://travis-ci.org/mapbox/mapbox-gl-native/builds/87963907

Got debug output. Tiles are getting stuck in the partial state.

INFO (General): --------------------------------------------------------------------------------
INFO (General): MapContext::styleURL: 
INFO (General): Source::id: mapbox
INFO (General): Source::loaded: 1
INFO (General): TileData::id: 0/0/0
INFO (General): TileData::state: parsed
INFO (General): TileData::error: 
INFO (General): Source::id: com.mapbox.annotations
INFO (General): Source::loaded: 1
INFO (General): TileData::id: 0/0/0
INFO (General): TileData::state: partial
INFO (General): TileData::error: 
INFO (General): Sprite::loaded: 1
INFO (General): --------------------------------------------------------------------------------

[jfirebaugh]

As of c0c554e, annotation ("Live") tiles have a hard time getting out of the partial state, because LiveTileData::parsePending calls worker.parseLiveTile, and never worker.parsePendingVectorTileLayers.

In the headless tests, the most common reason for an annotation tile to be pending is that the sprite hasn't been loaded. Once that condition is hit, there's no way for the annotation tile to get out of the pending state. The only way would be if invalidateTiles() was called, and that only happens if there are annotation changes.

#2792 is probably the best way of fixing this.

[jfirebaugh]

@tmpsantos I'm going to need to have you look at this. I thought 5df0928 would fix it, but it didn't.