Crashes in http_request_nsurl.mm

[webventil]

My iOS app often crashes in http_request_nsurl.mm. Unfortunately I can't say how to reproduce. Most of the times, it happens after going with the app back to foreground or dismissing a view controller with a map view.

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  39

Filtered syslog:
None found

Global Trace Buffer (reverse chronological seconds):
1.184214     CFNetwork                  0x000000018330d9a4 TCP Conn 0x14bd529b0 SSL Handshake DONE
1.404764     CFNetwork                  0x000000018330d8b4 TCP Conn 0x14bd529b0 starting SSL negotiation
1.405079     CFNetwork                  0x00000001833af760 TCP Conn 0x14bd529b0 complete. fd: 90, err: 0
1.405486     CFNetwork                  0x00000001833b0c8c TCP Conn 0x14bd529b0 event 1. err: 0
1.569433     CFNetwork                  0x00000001833b0d64 TCP Conn 0x14bd529b0 started
1.573233     CFNetwork                  0x000000018330d9a4 TCP Conn 0x14bbc2bd0 SSL Handshake DONE
1.652946     CFNetwork                  0x0000000183380f24 NSURLSessionTask finished with error - code: -999
1.653215     CFNetwork                  0x0000000183380f24 NSURLSessionTask finished with error - code: -999
1.920458     CFNetwork                  0x000000018330d8b4 TCP Conn 0x14bbc2bd0 starting SSL negotiation
1.920804     CFNetwork                  0x00000001833af760 TCP Conn 0x14bbc2bd0 complete. fd: 88, err: 0
1.922217     CFNetwork                  0x00000001833b0c8c TCP Conn 0x14bbc2bd0 event 1. err: 0
2.018634     CFNetwork                  0x00000001833b0d64 TCP Conn 0x14bbc2bd0 started
2.019849     CFNetwork                  0x0000000183380f24 NSURLSessionTask finished with error - code: -999
866.758107   CFNetwork                  0x000000018330d9a4 TCP Conn 0x14b8bf750 SSL Handshake DONE
867.026752   CFNetwork                  0x000000018330d8b4 TCP Conn 0x14b8bf750 starting SSL negotiation
867.027751   CFNetwork                  0x00000001833af760 TCP Conn 0x14b8bf750 complete. fd: 92, err: 0
867.028485   CFNetwork                  0x00000001833b0c8c TCP Conn 0x14b8bf750 event 1. err: 0
867.362534   CFNetwork                  0x00000001833b0d64 TCP Conn 0x14b8bf750 started
867.385020   CFNetwork                  0x000000018330d9a4 TCP Conn 0x14bb7b450 SSL Handshake DONE
867.416085   CFNetwork                  0x000000018330d8b4 TCP Conn 0x14bb7b450 starting SSL negotiation
867.416597   CFNetwork                  0x00000001833af760 TCP Conn 0x14bb7b450 complete. fd: 90, err: 0
867.416790   CFNetwork                  0x00000001833b0c8c TCP Conn 0x14bb7b450 event 1. err: 0
867.457173   CFNetwork                  0x00000001833b0d64 TCP Conn 0x14bb7b450 started
867.458879   CFNetwork                  0x000000018330d9a4 TCP Conn 0x14bb4fa80 SSL Handshake DONE
867.553220   CFNetwork                  0x000000018330d8b4 TCP Conn 0x14bb4fa80 starting SSL negotiation

Thread 39 name:  Dispatch queue: NSOperationQueue 0x14b958620 :: NSOperation 0x14bd72c10 (QOS: LEGACY)
Thread 39 Crashed:
0   libsystem_kernel.dylib          0x00000001991431e0 __pthread_kill + 8
1   libsystem_pthread.dylib         0x000000019920cf0c pthread_kill + 112
2   libsystem_c.dylib               0x00000001990b6b78 abort + 140
3   libsystem_malloc.dylib          0x0000000199170384 szone_size + 0
4   Mapbox                          0x000000010080dea8 mbgl::HTTPNSURLRequest::handleResult(NSData*, NSURLResponse*, NSError*) (in Mapbox) (http_request_nsurl.mm:256) + 1056424
5   CFNetwork                       0x000000018331749c __75-[__NSURLSessionLocal taskForClass:request:uploadFile:bodyData:completion:]_block_invoke + 32
6   CFNetwork                       0x000000018332a454 __49-[__NSCFLocalSessionTask _task_onqueue_didFinish]_block_invoke + 296
7   Foundation                      0x0000000184b2b82c __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 16
8   Foundation                      0x0000000184a7e6c0 -[NSBlockOperation main] + 96
9   Foundation                      0x0000000184a6e918 -[__NSOperationInternal _start:] + 604
10  Foundation                      0x0000000184b2dc20 __NSOQSchedule_f + 224
11  libdispatch.dylib               0x0000000198ff5770 _dispatch_client_callout + 16
12  libdispatch.dylib               0x000000019900175c _dispatch_queue_drain + 864
13  libdispatch.dylib               0x0000000198ff9274 _dispatch_queue_invoke + 464
14  libdispatch.dylib               0x000000019900362c _dispatch_root_queue_drain + 728
15  libdispatch.dylib               0x000000019900334c _dispatch_worker_thread3 + 112
16  libsystem_pthread.dylib         0x0000000199209478 _pthread_wqthread + 1092
17  libsystem_pthread.dylib         0x0000000199209028 start_wqthread + 4

[ljbade]

Is this related to the uv_async issues? #2400

[webventil]

Probably yes. Today I got another crash in uv_async. This occurs often for me when creating a new map view instance in poor network conditions e.g. during the switch from wifi to cellular network or vice versa.

Thread 29 name:  Dispatch queue: NSOperationQueue 0x14758cf10 :: NSOperation 0x14b387430 (QOS: LEGACY)
Thread 29 Crashed:
0   Mapbox                          0x00000001007eb7d0 uv__async_make_pending (in Mapbox) (async.c:87) + 1062864
1   Mapbox                          0x00000001007eb79c uv_async_send (in Mapbox) (async.c:57) + 1062812
2   Mapbox                          0x00000001007ea304 mbgl::HTTPNSURLRequest::handleResult(NSData*, NSURLResponse*, NSError*) (in Mapbox) (uv_detail.hpp:116) + 1057540
3   CFNetwork                       0x000000018331749c __75-[__NSURLSessionLocal taskForClass:request:uploadFile:bodyData:completion:]_block_invoke + 32
4   CFNetwork                       0x000000018332a454 __49-[__NSCFLocalSessionTask _task_onqueue_didFinish]_block_invoke + 296
5   Foundation                      0x0000000184b2b82c __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 16
6   Foundation                      0x0000000184a7e6c0 -[NSBlockOperation main] + 96
7   Foundation                      0x0000000184a6e918 -[__NSOperationInternal _start:] + 604
8   Foundation                      0x0000000184b2dc20 __NSOQSchedule_f + 224
9   libdispatch.dylib               0x0000000198ff5770 _dispatch_client_callout + 16
10  libdispatch.dylib               0x000000019900175c _dispatch_queue_drain + 864
11  libdispatch.dylib               0x0000000198ff9274 _dispatch_queue_invoke + 464
12  libdispatch.dylib               0x000000019900362c _dispatch_root_queue_drain + 728
13  libdispatch.dylib               0x000000019900334c _dispatch_worker_thread3 + 112
14  libsystem_pthread.dylib         0x0000000199209478 _pthread_wqthread + 1092
15  libsystem_pthread.dylib         0x0000000199209028 start_wqthread + 4

[picciano]

We're seeing this one a lot too. It affects both iOS 8 and 9 and across different devices and users.

[webventil]

The amount of crashes decreases for me, if i turn of the new iOS9 Wi-Fi Assist. This seems to be a problem.

[ljbade]

@kkaefer Have you had any luck on narrowing down the uv_async issues? Is this possibly a bug in libuv itself?

[1ec5]

If turning off Wifi Assist helps, perhaps Reachability callbacks have something to do with reproducing the issue directly.

[ljbade]

@1ec5 does Reachability cause uv_async to be called more frequently so it just crashes faster? Hmm...

[1ec5]

Don’t mind uv_async(). The bug is likely in mbgl::HTTPNSURLRequest::handleResult().

[ljbade]

@1ec5 - interesting the Android OkHTTP implementation was based on the NSURL one (ref #2400) so it is likely I copied the same bug whatever it is.

[picciano]

Looks like the same issue, but slightly different stack trace.

Thread : Crashed: NSOperationQueue 0x15884ac20 :: NSOperation 0x156512530 (QOS: LEGACY)
0  TallyGo                        0x000000010036a684 uv_async_send (async.c:62)
1  TallyGo                        0x00000001003685a0 mbgl::HTTPNSURLRequest::handleResult(NSData*, NSURLResponse*, NSError*) (uv_detail.hpp:124)
2  TallyGo                        0x00000001003685a0 mbgl::HTTPNSURLRequest::handleResult(NSData*, NSURLResponse*, NSError*) (uv_detail.hpp:124)
3  CFNetwork                      0x0000000181d3749c __75-[__NSURLSessionLocal taskForClass:request:uploadFile:bodyData:completion:]_block_invoke + 32
4  CFNetwork                      0x0000000181d4a454 __49-[__NSCFLocalSessionTask _task_onqueue_didFinish]_block_invoke + 296
5  Foundation                     0x000000018354b82c __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 16
6  Foundation                     0x000000018349e6c0 -[NSBlockOperation main] + 96
7  Foundation                     0x000000018348e918 -[__NSOperationInternal _start:] + 604
8  Foundation                     0x000000018354dc20 __NSOQSchedule_f + 224
9  libdispatch.dylib              0x0000000197a19770 _dispatch_client_callout + 16
10 libdispatch.dylib              0x0000000197a2575c _dispatch_queue_drain + 864
11 libdispatch.dylib              0x0000000197a1d274 _dispatch_queue_invoke + 464
12 libdispatch.dylib              0x0000000197a2762c _dispatch_root_queue_drain + 728
13 libdispatch.dylib              0x0000000197a2734c _dispatch_worker_thread3 + 112
14 libsystem_pthread.dylib        0x0000000197c2d478 _pthread_wqthread + 1092
15 libsystem_pthread.dylib        0x0000000197c2d028 start_wqthread + 4

[minikin]

I also have the crash related to mbgl::HTTPNSURLRequest:

Crashed: NSOperationQueue 0x15d66580 :: NSOperation 0x17f85670 (QOS: LEGACY)
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x000000000000002e

Thread : Crashed: NSOperationQueue 0x15d66580 :: NSOperation 0x17f85670 (QOS: LEGACY)
0  libc++.1.dylib                 904587130 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::~basic_string() + 5
1  Mapbox                         4972305 mbgl::HTTPNSURLRequest::handleResult(NSData*, NSURLResponse*, NSError*) + 280
2  Mapbox                         4972305 mbgl::HTTPNSURLRequest::handleResult(NSData*, NSURLResponse*, NSError*) + 280
3  CFNetwork                      609895861 __75-[__NSURLSessionLocal taskForClass:request:uploadFile:bodyData:completion:]_block_invoke + 16
4  CFNetwork                      609959127 __49-[__NSCFLocalSessionTask _task_onqueue_didFinish]_block_invoke + 278
5  Foundation                     632538477 __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 8
6  Foundation                     631889887 -[NSBlockOperation main] + 146
7  Foundation                     631833551 -[__NSOperationInternal _start:] + 774
8  Foundation                     632547373 __NSOQSchedule_f + 192

[kkaefer]

@webventil @minikin I just merged #2617 and can't reproduce this anymore. Can you please check out the current master build and see if that issue still exists?

[kkaefer]

I'm attributing this crash to #2123. Please reopen if you continue to observe this issue after #2123 is merged.