__info__.py 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291
  1. #!/usr/bin/env python
  2. #
  3. # Author: Mike McKerns (mmckerns @caltech and @uqfoundation)
  4. # Copyright (c) 2024 The Uncertainty Quantification Foundation.
  5. # License: 3-clause BSD. The full license text is available at:
  6. # - https://github.com/uqfoundation/dill/blob/master/LICENSE
  7. '''
  8. -----------------------------
  9. dill: serialize all of Python
  10. -----------------------------
  11. About Dill
  12. ==========
  13. ``dill`` extends Python's ``pickle`` module for serializing and de-serializing
  14. Python objects to the majority of the built-in Python types. Serialization
  15. is the process of converting an object to a byte stream, and the inverse
  16. of which is converting a byte stream back to a Python object hierarchy.
  17. ``dill`` provides the user the same interface as the ``pickle`` module, and
  18. also includes some additional features. In addition to pickling Python
  19. objects, ``dill`` provides the ability to save the state of an interpreter
  20. session in a single command. Hence, it would be feasible to save an
  21. interpreter session, close the interpreter, ship the pickled file to
  22. another computer, open a new interpreter, unpickle the session and
  23. thus continue from the 'saved' state of the original interpreter
  24. session.
  25. ``dill`` can be used to store Python objects to a file, but the primary
  26. usage is to send Python objects across the network as a byte stream.
  27. ``dill`` is quite flexible, and allows arbitrary user defined classes
  28. and functions to be serialized. Thus ``dill`` is not intended to be
  29. secure against erroneously or maliciously constructed data. It is
  30. left to the user to decide whether the data they unpickle is from
  31. a trustworthy source.
  32. ``dill`` is part of ``pathos``, a Python framework for heterogeneous computing.
  33. ``dill`` is in active development, so any user feedback, bug reports, comments,
  34. or suggestions are highly appreciated. A list of issues is located at
  35. https://github.com/uqfoundation/dill/issues, with a legacy list maintained at
  36. https://uqfoundation.github.io/project/pathos/query.
  37. Major Features
  38. ==============
  39. ``dill`` can pickle the following standard types:
  40. - none, type, bool, int, float, complex, bytes, str,
  41. - tuple, list, dict, file, buffer, builtin,
  42. - Python classes, namedtuples, dataclasses, metaclasses,
  43. - instances of classes,
  44. - set, frozenset, array, functions, exceptions
  45. ``dill`` can also pickle more 'exotic' standard types:
  46. - functions with yields, nested functions, lambdas,
  47. - cell, method, unboundmethod, module, code, methodwrapper,
  48. - methoddescriptor, getsetdescriptor, memberdescriptor, wrapperdescriptor,
  49. - dictproxy, slice, notimplemented, ellipsis, quit
  50. ``dill`` cannot yet pickle these standard types:
  51. - frame, generator, traceback
  52. ``dill`` also provides the capability to:
  53. - save and load Python interpreter sessions
  54. - save and extract the source code from functions and classes
  55. - interactively diagnose pickling errors
  56. Current Release
  57. ===============
  58. The latest released version of ``dill`` is available from:
  59. https://pypi.org/project/dill
  60. ``dill`` is distributed under a 3-clause BSD license.
  61. Development Version
  62. ===================
  63. You can get the latest development version with all the shiny new features at:
  64. https://github.com/uqfoundation
  65. If you have a new contribution, please submit a pull request.
  66. Installation
  67. ============
  68. ``dill`` can be installed with ``pip``::
  69. $ pip install dill
  70. To optionally include the ``objgraph`` diagnostic tool in the install::
  71. $ pip install dill[graph]
  72. To optionally include the ``gprof2dot`` diagnostic tool in the install::
  73. $ pip install dill[profile]
  74. For windows users, to optionally install session history tools::
  75. $ pip install dill[readline]
  76. Requirements
  77. ============
  78. ``dill`` requires:
  79. - ``python`` (or ``pypy``), **>=3.8**
  80. - ``setuptools``, **>=42**
  81. Optional requirements:
  82. - ``objgraph``, **>=1.7.2**
  83. - ``gprof2dot``, **>=2022.7.29**
  84. - ``pyreadline``, **>=1.7.1** (on windows)
  85. Basic Usage
  86. ===========
  87. ``dill`` is a drop-in replacement for ``pickle``. Existing code can be
  88. updated to allow complete pickling using::
  89. >>> import dill as pickle
  90. or::
  91. >>> from dill import dumps, loads
  92. ``dumps`` converts the object to a unique byte string, and ``loads`` performs
  93. the inverse operation::
  94. >>> squared = lambda x: x**2
  95. >>> loads(dumps(squared))(3)
  96. 9
  97. There are a number of options to control serialization which are provided
  98. as keyword arguments to several ``dill`` functions:
  99. * with *protocol*, the pickle protocol level can be set. This uses the
  100. same value as the ``pickle`` module, *DEFAULT_PROTOCOL*.
  101. * with *byref=True*, ``dill`` to behave a lot more like pickle with
  102. certain objects (like modules) pickled by reference as opposed to
  103. attempting to pickle the object itself.
  104. * with *recurse=True*, objects referred to in the global dictionary are
  105. recursively traced and pickled, instead of the default behavior of
  106. attempting to store the entire global dictionary.
  107. * with *fmode*, the contents of the file can be pickled along with the file
  108. handle, which is useful if the object is being sent over the wire to a
  109. remote system which does not have the original file on disk. Options are
  110. *HANDLE_FMODE* for just the handle, *CONTENTS_FMODE* for the file content
  111. and *FILE_FMODE* for content and handle.
  112. * with *ignore=False*, objects reconstructed with types defined in the
  113. top-level script environment use the existing type in the environment
  114. rather than a possibly different reconstructed type.
  115. The default serialization can also be set globally in *dill.settings*.
  116. Thus, we can modify how ``dill`` handles references to the global dictionary
  117. locally or globally::
  118. >>> import dill.settings
  119. >>> dumps(absolute) == dumps(absolute, recurse=True)
  120. False
  121. >>> dill.settings['recurse'] = True
  122. >>> dumps(absolute) == dumps(absolute, recurse=True)
  123. True
  124. ``dill`` also includes source code inspection, as an alternate to pickling::
  125. >>> import dill.source
  126. >>> print(dill.source.getsource(squared))
  127. squared = lambda x:x**2
  128. To aid in debugging pickling issues, use *dill.detect* which provides
  129. tools like pickle tracing::
  130. >>> import dill.detect
  131. >>> with dill.detect.trace():
  132. >>> dumps(squared)
  133. ┬ F1: <function <lambda> at 0x7fe074f8c280>
  134. ├┬ F2: <function _create_function at 0x7fe074c49c10>
  135. │└ # F2 [34 B]
  136. ├┬ Co: <code object <lambda> at 0x7fe07501eb30, file "<stdin>", line 1>
  137. │├┬ F2: <function _create_code at 0x7fe074c49ca0>
  138. ││└ # F2 [19 B]
  139. │└ # Co [87 B]
  140. ├┬ D1: <dict object at 0x7fe0750d4680>
  141. │└ # D1 [22 B]
  142. ├┬ D2: <dict object at 0x7fe074c5a1c0>
  143. │└ # D2 [2 B]
  144. ├┬ D2: <dict object at 0x7fe074f903c0>
  145. │├┬ D2: <dict object at 0x7fe074f8ebc0>
  146. ││└ # D2 [2 B]
  147. │└ # D2 [23 B]
  148. └ # F1 [180 B]
  149. With trace, we see how ``dill`` stored the lambda (``F1``) by first storing
  150. ``_create_function``, the underlying code object (``Co``) and ``_create_code``
  151. (which is used to handle code objects), then we handle the reference to
  152. the global dict (``D2``) plus other dictionaries (``D1`` and ``D2``) that
  153. save the lambda object's state. A ``#`` marks when the object is actually stored.
  154. More Information
  155. ================
  156. Probably the best way to get started is to look at the documentation at
  157. http://dill.rtfd.io. Also see ``dill.tests`` for a set of scripts that
  158. demonstrate how ``dill`` can serialize different Python objects. You can
  159. run the test suite with ``python -m dill.tests``. The contents of any
  160. pickle file can be examined with ``undill``. As ``dill`` conforms to
  161. the ``pickle`` interface, the examples and documentation found at
  162. http://docs.python.org/library/pickle.html also apply to ``dill``
  163. if one will ``import dill as pickle``. The source code is also generally
  164. well documented, so further questions may be resolved by inspecting the
  165. code itself. Please feel free to submit a ticket on github, or ask a
  166. question on stackoverflow (**@Mike McKerns**).
  167. If you would like to share how you use ``dill`` in your work, please send
  168. an email (to **mmckerns at uqfoundation dot org**).
  169. Citation
  170. ========
  171. If you use ``dill`` to do research that leads to publication, we ask that you
  172. acknowledge use of ``dill`` by citing the following in your publication::
  173. M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
  174. "Building a framework for predictive science", Proceedings of
  175. the 10th Python in Science Conference, 2011;
  176. http://arxiv.org/pdf/1202.1056
  177. Michael McKerns and Michael Aivazis,
  178. "pathos: a framework for heterogeneous computing", 2010- ;
  179. https://uqfoundation.github.io/project/pathos
  180. Please see https://uqfoundation.github.io/project/pathos or
  181. http://arxiv.org/pdf/1202.1056 for further information.
  182. '''
  183. __version__ = '0.3.9'
  184. __author__ = 'Mike McKerns'
  185. __license__ = '''
  186. Copyright (c) 2004-2016 California Institute of Technology.
  187. Copyright (c) 2016-2024 The Uncertainty Quantification Foundation.
  188. All rights reserved.
  189. This software is available subject to the conditions and terms laid
  190. out below. By downloading and using this software you are agreeing
  191. to the following conditions.
  192. Redistribution and use in source and binary forms, with or without
  193. modification, are permitted provided that the following conditions
  194. are met:
  195. - Redistributions of source code must retain the above copyright
  196. notice, this list of conditions and the following disclaimer.
  197. - Redistributions in binary form must reproduce the above copyright
  198. notice, this list of conditions and the following disclaimer in the
  199. documentation and/or other materials provided with the distribution.
  200. - Neither the names of the copyright holders nor the names of any of
  201. the contributors may be used to endorse or promote products derived
  202. from this software without specific prior written permission.
  203. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
  204. "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
  205. TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
  206. PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
  207. CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
  208. EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
  209. PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
  210. OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
  211. WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
  212. OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
  213. ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  214. '''