123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291 |
- #!/usr/bin/env python
- #
- # Author: Mike McKerns (mmckerns @caltech and @uqfoundation)
- # Copyright (c) 2024 The Uncertainty Quantification Foundation.
- # License: 3-clause BSD. The full license text is available at:
- # - https://github.com/uqfoundation/dill/blob/master/LICENSE
- '''
- -----------------------------
- dill: serialize all of Python
- -----------------------------
- About Dill
- ==========
- ``dill`` extends Python's ``pickle`` module for serializing and de-serializing
- Python objects to the majority of the built-in Python types. Serialization
- is the process of converting an object to a byte stream, and the inverse
- of which is converting a byte stream back to a Python object hierarchy.
- ``dill`` provides the user the same interface as the ``pickle`` module, and
- also includes some additional features. In addition to pickling Python
- objects, ``dill`` provides the ability to save the state of an interpreter
- session in a single command. Hence, it would be feasible to save an
- interpreter session, close the interpreter, ship the pickled file to
- another computer, open a new interpreter, unpickle the session and
- thus continue from the 'saved' state of the original interpreter
- session.
- ``dill`` can be used to store Python objects to a file, but the primary
- usage is to send Python objects across the network as a byte stream.
- ``dill`` is quite flexible, and allows arbitrary user defined classes
- and functions to be serialized. Thus ``dill`` is not intended to be
- secure against erroneously or maliciously constructed data. It is
- left to the user to decide whether the data they unpickle is from
- a trustworthy source.
- ``dill`` is part of ``pathos``, a Python framework for heterogeneous computing.
- ``dill`` is in active development, so any user feedback, bug reports, comments,
- or suggestions are highly appreciated. A list of issues is located at
- https://github.com/uqfoundation/dill/issues, with a legacy list maintained at
- https://uqfoundation.github.io/project/pathos/query.
- Major Features
- ==============
- ``dill`` can pickle the following standard types:
- - none, type, bool, int, float, complex, bytes, str,
- - tuple, list, dict, file, buffer, builtin,
- - Python classes, namedtuples, dataclasses, metaclasses,
- - instances of classes,
- - set, frozenset, array, functions, exceptions
- ``dill`` can also pickle more 'exotic' standard types:
- - functions with yields, nested functions, lambdas,
- - cell, method, unboundmethod, module, code, methodwrapper,
- - methoddescriptor, getsetdescriptor, memberdescriptor, wrapperdescriptor,
- - dictproxy, slice, notimplemented, ellipsis, quit
- ``dill`` cannot yet pickle these standard types:
- - frame, generator, traceback
- ``dill`` also provides the capability to:
- - save and load Python interpreter sessions
- - save and extract the source code from functions and classes
- - interactively diagnose pickling errors
- Current Release
- ===============
- The latest released version of ``dill`` is available from:
- https://pypi.org/project/dill
- ``dill`` is distributed under a 3-clause BSD license.
- Development Version
- ===================
- You can get the latest development version with all the shiny new features at:
- https://github.com/uqfoundation
- If you have a new contribution, please submit a pull request.
- Installation
- ============
- ``dill`` can be installed with ``pip``::
- $ pip install dill
- To optionally include the ``objgraph`` diagnostic tool in the install::
- $ pip install dill[graph]
- To optionally include the ``gprof2dot`` diagnostic tool in the install::
- $ pip install dill[profile]
- For windows users, to optionally install session history tools::
- $ pip install dill[readline]
- Requirements
- ============
- ``dill`` requires:
- - ``python`` (or ``pypy``), **>=3.8**
- - ``setuptools``, **>=42**
- Optional requirements:
- - ``objgraph``, **>=1.7.2**
- - ``gprof2dot``, **>=2022.7.29**
- - ``pyreadline``, **>=1.7.1** (on windows)
- Basic Usage
- ===========
- ``dill`` is a drop-in replacement for ``pickle``. Existing code can be
- updated to allow complete pickling using::
- >>> import dill as pickle
- or::
- >>> from dill import dumps, loads
- ``dumps`` converts the object to a unique byte string, and ``loads`` performs
- the inverse operation::
- >>> squared = lambda x: x**2
- >>> loads(dumps(squared))(3)
- 9
- There are a number of options to control serialization which are provided
- as keyword arguments to several ``dill`` functions:
- * with *protocol*, the pickle protocol level can be set. This uses the
- same value as the ``pickle`` module, *DEFAULT_PROTOCOL*.
- * with *byref=True*, ``dill`` to behave a lot more like pickle with
- certain objects (like modules) pickled by reference as opposed to
- attempting to pickle the object itself.
- * with *recurse=True*, objects referred to in the global dictionary are
- recursively traced and pickled, instead of the default behavior of
- attempting to store the entire global dictionary.
- * with *fmode*, the contents of the file can be pickled along with the file
- handle, which is useful if the object is being sent over the wire to a
- remote system which does not have the original file on disk. Options are
- *HANDLE_FMODE* for just the handle, *CONTENTS_FMODE* for the file content
- and *FILE_FMODE* for content and handle.
- * with *ignore=False*, objects reconstructed with types defined in the
- top-level script environment use the existing type in the environment
- rather than a possibly different reconstructed type.
- The default serialization can also be set globally in *dill.settings*.
- Thus, we can modify how ``dill`` handles references to the global dictionary
- locally or globally::
- >>> import dill.settings
- >>> dumps(absolute) == dumps(absolute, recurse=True)
- False
- >>> dill.settings['recurse'] = True
- >>> dumps(absolute) == dumps(absolute, recurse=True)
- True
- ``dill`` also includes source code inspection, as an alternate to pickling::
- >>> import dill.source
- >>> print(dill.source.getsource(squared))
- squared = lambda x:x**2
- To aid in debugging pickling issues, use *dill.detect* which provides
- tools like pickle tracing::
- >>> import dill.detect
- >>> with dill.detect.trace():
- >>> dumps(squared)
- ┬ F1: <function <lambda> at 0x7fe074f8c280>
- ├┬ F2: <function _create_function at 0x7fe074c49c10>
- │└ # F2 [34 B]
- ├┬ Co: <code object <lambda> at 0x7fe07501eb30, file "<stdin>", line 1>
- │├┬ F2: <function _create_code at 0x7fe074c49ca0>
- ││└ # F2 [19 B]
- │└ # Co [87 B]
- ├┬ D1: <dict object at 0x7fe0750d4680>
- │└ # D1 [22 B]
- ├┬ D2: <dict object at 0x7fe074c5a1c0>
- │└ # D2 [2 B]
- ├┬ D2: <dict object at 0x7fe074f903c0>
- │├┬ D2: <dict object at 0x7fe074f8ebc0>
- ││└ # D2 [2 B]
- │└ # D2 [23 B]
- └ # F1 [180 B]
- With trace, we see how ``dill`` stored the lambda (``F1``) by first storing
- ``_create_function``, the underlying code object (``Co``) and ``_create_code``
- (which is used to handle code objects), then we handle the reference to
- the global dict (``D2``) plus other dictionaries (``D1`` and ``D2``) that
- save the lambda object's state. A ``#`` marks when the object is actually stored.
- More Information
- ================
- Probably the best way to get started is to look at the documentation at
- http://dill.rtfd.io. Also see ``dill.tests`` for a set of scripts that
- demonstrate how ``dill`` can serialize different Python objects. You can
- run the test suite with ``python -m dill.tests``. The contents of any
- pickle file can be examined with ``undill``. As ``dill`` conforms to
- the ``pickle`` interface, the examples and documentation found at
- http://docs.python.org/library/pickle.html also apply to ``dill``
- if one will ``import dill as pickle``. The source code is also generally
- well documented, so further questions may be resolved by inspecting the
- code itself. Please feel free to submit a ticket on github, or ask a
- question on stackoverflow (**@Mike McKerns**).
- If you would like to share how you use ``dill`` in your work, please send
- an email (to **mmckerns at uqfoundation dot org**).
- Citation
- ========
- If you use ``dill`` to do research that leads to publication, we ask that you
- acknowledge use of ``dill`` by citing the following in your publication::
- M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
- "Building a framework for predictive science", Proceedings of
- the 10th Python in Science Conference, 2011;
- http://arxiv.org/pdf/1202.1056
- Michael McKerns and Michael Aivazis,
- "pathos: a framework for heterogeneous computing", 2010- ;
- https://uqfoundation.github.io/project/pathos
- Please see https://uqfoundation.github.io/project/pathos or
- http://arxiv.org/pdf/1202.1056 for further information.
- '''
- __version__ = '0.3.9'
- __author__ = 'Mike McKerns'
- __license__ = '''
- Copyright (c) 2004-2016 California Institute of Technology.
- Copyright (c) 2016-2024 The Uncertainty Quantification Foundation.
- All rights reserved.
- This software is available subject to the conditions and terms laid
- out below. By downloading and using this software you are agreeing
- to the following conditions.
- Redistribution and use in source and binary forms, with or without
- modification, are permitted provided that the following conditions
- are met:
- - Redistributions of source code must retain the above copyright
- notice, this list of conditions and the following disclaimer.
- - Redistributions in binary form must reproduce the above copyright
- notice, this list of conditions and the following disclaimer in the
- documentation and/or other materials provided with the distribution.
- - Neither the names of the copyright holders nor the names of any of
- the contributors may be used to endorse or promote products derived
- from this software without specific prior written permission.
- THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
- TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
- CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
- EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
- PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
- OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
- WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
- OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
- ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- '''
|