ARROW-1758: [Python] Remove pickle=True option for object serialization

[Licht-T]

This closes ARROW-1758.

[wesm]

If no custom serializer/deserializer is passed, should this default to pickle?

[pcmoritz]

Currently it defaults to using __dict__, which is more efficient so it seems a good default, we use that in Ray for pretty much all types (however rarely I have seen cases where it doesn't work). I'd prefer to keep __dict__ the default but no big deals since it can easily be changed.

[wesm]

What about lambdas?

[pcmoritz]

In Ray, lambdas are handled with cloudpickle as are builtin types like "type"; most user defined types are handled via dict; namedtuples are special cased.

One reasonable way to handle this is to provide default callbacks for types we encounter that don't work with dict (using cloudpickle or a custom serializer/deserializer), there are already examples of this in the repo, and use dict for anything else. That worked well for us so far. If this strategy doesn't work for the user, they have full flexibility by providing their own serialization context.

What do you think?

[wesm]

This is sufficiently advanced stuff that I think it's not unreasonable to ask people to explicitly use cloudpickle in a case like this. I would be OK if we changed this patch to do that to get the build passing, and we can always move on later

[wesm]

The tests fail here, need to use cloudpickle, I think

[Licht-T]

Okay, I'll check.

[wesm]

@pcmoritz @robertnishihara does this look ok?

[robertnishihara]

Looks good to me.

[pcmoritz]

+1 LGTM

[Licht-T]

Thanks @wesm!