AttributeError when pickled function uses submodules

This problem is when a function refers (by attribute) to a sub-module of a package. Cloudpickle appears to pickle functions not by name, but by code plus (a subset of) globals. So the parent package is injected into the pickle, but the sub-module is not.

def func():
    # import unittest.mock
    x = unittest.TestCase
    x = unittest.mock.Mock
import unittest.mock

import cloudpickle as pickle
s = pickle.dumps(func)

del unittest
import sys
del sys.modules['unittest']
del sys.modules['unittest.mock']

f = pickle.loads(s)
# import unittest.mock as anything
f()
AttributeError: module 'unittest' has no attribute 'mock'

This leads to non-intuitive bugs in applications such as cluster computing (e.g. with dask.distributed).
Workarounds:

perform imports inside functions (contrary to PEP8).
import sub-modules (or their contents) as globals.
ensure (somehow) that all relevant sub-modules are automatically imported by respective parent packages (e.g. __init__.py).
arrange for the unpickling process to have already done an import of the sub-module (e.g. by first uncloudpickling something else that did refer to the sub-module as a global).

I assume cloudpickle checks whether a global is an imported module, and if so then stores the name (rather than pickling its attributes). Is it practical to also check (via sys.modules.keys()) which sub-modules had previously been imported, and ensure every such module is subsequently initialised?

For example, could modify CloudPickler.save_module to include

for name in sys.modules.keys():
    if name.startswith(obj.__name__ + '.'):
        self.save_reduce(subimport, (name,) ...

Alternatively, since the above might be overzealous, could instead in CloudPickler.extract_func_data do something roughly like replace f_globals = {} with:

f_globals = {k:v for k,v in sys.modules.items() 
             if not set(k.split('.')) - set(func.__code__.co_names)}

I'm thinking along the lines of a change to cloudpickle.CloudPickler.save_function_tuple.

import cloudpickle

import unittest.mock
unpatched = cloudpickle.CloudPickler.extract_func_data                  
def picklepatch(self, func):    
    code, f_globals, defaults, closure, dct, base_globals = unpatched(self, func)
    
    # check if any top-level dependency is an imported package
    for x in set(f_globals.values()) | set(closure): 
        if isinstance(x, cloudpickle.types.ModuleType) and x.__package__:
            # check if the package has any currently loaded sub-imports
            prefix = x.__name__ + '.'
            for name, module in cloudpickle.sys.modules.items():
                if name.startswith(prefix): 
                    # check whether the function can address the sub-module
                    tokens = set(name[len(prefix):].split('.'))
                    if not tokens - set(code.co_names):
                        self.save(module) # ensure the unpickler executes import of this submodule
                        self.write(cloudpickle.pickle.POP) # then discard the reference to it
        
    return code, f_globals, defaults, closure, dct, base_globals
patch = unittest.mock.patch('cloudpickle.CloudPickler.extract_func_data', new=picklepatch)


#def scope():
#    import http.cookies
#    def example():
#        x = http.HTTPStatus
#        x = http.cookies.Morsel
#        print('test successful')
#    return example
#example = scope()

import http.cookies
def example():
    x = http.HTTPStatus
    x = http.cookies.Morsel
    print('test successful')

broken = cloudpickle.dumps(example)
with patch:
    fixed = cloudpickle.dumps(example)

# refresh the environment (unimport http.cookies)
del http
del cloudpickle.sys.modules['http.cookies']
del cloudpickle.sys.modules['http']

# deserialise
f = cloudpickle.loads(broken)
try:
    f()
except AttributeError:
    print('patch required')    
g = cloudpickle.loads(fixed)
g()

Add regression tests for submodule dependence of pickled function, is…

save submodules to resolve issue cloudpipe#78

Add regression tests for submodule dependence of pickled function, is…

save submodules to resolve issue cloudpipe#78

This is fixed in the latest release

Navigation Menu

Repository navigation

AttributeError when pickled function uses submodules #78

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AttributeError when pickled function uses submodules #78

Description

Activity

benjimin commented on Feb 1, 2017

benjimin commented on Feb 7, 2017

llllllllll commented on May 31, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions