C++博客 :: 首页 :: 新随笔 :: 联系 :: 聚合  :: 管理 ::
  379 随笔 :: 0 文章 :: 454 评论 :: 0 Trackbacks

Programming Python, 3rd Edition 翻译

19.4. Pickled Objects

19.4. Pickle对象

Probably the biggest limitation of DBM keyed files is in what they can store: data stored under a key must be a simple text string. If you want to store Python objects in a DBM file, you can sometimes manually convert them to and from strings on writes and reads (e.g., with str and eval calls), but this takes you only so far. For arbitrarily complex Python objects such as class instances and nested data structures, you need something more. Class instance objects, for example, cannot be later re-created from their standard string representations. Custom to-string conversions are error prone and not general.

DBM 键控文件(DBM keyed file)最大的限制也许在于他们可以存储的东西:一个键值下存储的数据必须是个简单文本字符串。如果您想要在DBM文件中储存Python对象,有时您 可以在读写的时候,手动进行与字符串的转换(例如,用str和eval调用),但只能做到这样。对任意复杂的Python对象,如类实例和嵌套的数据结 构,您需要更多的东西。例如,类实例对象以后无法从其标准字符串表达(string representation)重建。自定义的到字符串的转换容易出错,并且不通用。

The Python pickle module, a standard part of the Python system, provides the conversion step needed. It converts nearly arbitrary Python in-memory objects to and from a single linear string format, suitable for storing in flat files, shipping across network sockets between trusted sources, and so on. This conversion from object to string is often called serializationarbitrary data structures in memory are mapped to a serial string form.

Python 系统的标准部件,pickle模块,提供了所需的转换步骤。它可以将几乎任意的Python内存对象,转换为单一线性的字符串格式,使之适于无格式文件存 储,或在可靠来源之间跨越网络套接口传输等等,并可反向转换。这种从对象到字符串的转换通常被称为序列化(serialization):将内存中的任意 数据结构映射为串行字符串形式。

The string representation used for objects is also sometimes referred to as a byte stream, due to its linear format. It retains all the content and references structure of the original in-memory object. When the object is later re-created from its byte string, it will be a new in-memory object identical in structure and value to the original, though located at a different memory address. The re-created object is effectively a copy of the original.


Pickling works on almost any Python datatypenumbers, lists, dictionaries, class instances, nested structures, and moreand so is a general way to store data. Because pickles contain native Python objects, there is almost no database API to be found; the objects stored are processed with normal Python syntax when they are later retrieved.


19.4.1. Using Object Pickling

19.4.1. 使用对象pickle

Pickling may sound complicated the first time you encounter it, but the good news is that Python hides all the complexity of object-to-string conversion. In fact, the pickle module 's interfaces are incredibly simple to use. For example, to pickle an object into a serialized string, we can either make a pickler and call its methods or use convenience functions in the module to achieve the same effect:

第 一次听到pickle,可能觉得有点复杂,但好消息是,Python隐藏了所有从对象到字符串转换的复杂性。事实上,pickle模块的接口简单易用,简 直令人难以置信。例如,要pickle对象到一个序列化字符串,我们可以生成一个pickler,并调用其方法,或使用模块中的便捷函数来达到相同的效 果:


P = pickle.Pickler( file) 

Make a new pickler for pickling to an open output file object file.


P.dump( object) 

Write an object onto the pickler's file/stream.


pickle.dump( object, file) 

Same as the last two calls combined: pickle an object onto an open file.


string = pickle.dumps( object) 

Return the pickled representation of object as a character string.

Unpickling from a serialized string back to the original object is similarboth object and convenience function interfaces are available:



U = pickle.Unpickler( file) 

Make an unpickler for unpickling from an open input file object file.
生成一个unpickler,用来从一个打开的文件对象file unpickle。


object = U.load( )

Read an object from the unpickler's file/stream.


object = pickle.load( file) 

Same as the last two calls combined: unpickle an object from an open file.


object = pickle.loads( string) 

Read an object from a character string rather than a file.

Pickler and Unpickler are exported classes. In all of the preceding cases, file is either an open file object or any object that implements the same attributes as file objects:


Pickler calls the file's write method with a string argument.


Unpickler calls the file's read method with a byte count, and readline without arguments.


Any object that provides these attributes can be passed in to the file parameters. In particular, file can be an instance of a Python class that provides the read/write methods (i.e., the expected file-like interface). This lets you map pickled streams to in-memory objects with classes, for arbitrary use. For instance, the StringIO standard library module discussed in Chapter 3 provides classes that map file calls to and from in-memory strings.

任 何提供这些属性的对象都可以作为file参数传入。特别是,file可以是一个提供了读/写方法的Python类实例(即预期的类似文件的接口)。这让您 可以用类映射pickle流到内存对象,并可任意使用。例如,第3章讨论的标准库模块StringIO提供的类,它们可映射文件调用到内存字符串或反之。

This hook also lets you ship Python objects across a network, by providing sockets wrapped to look like files in pickle calls at the sender, and unpickle calls at the receiver (see the sidebar "Making Sockets Look Like Files," in Chapter 13, for more details). In fact, for some, pickling Python objects across a trusted network serves as a simpler alternative to network transport protocols such as SOAP and XML-RPC; provided that Python is on both ends of the communication (pickled objects are represented with a Python-specific format, not with XML text).

该 挂钩也可以让您通过网络传输Python对象,只要封装套接口,使之看上去像发送端pickle调用中的文件,以及像接收端unpickle调用中的文件 (详见第13章侧栏“使套接口看上去像文件”)。事实上,对一些人来说,pickle Python对象并在一个值得信赖的网络上传输,是替代如SOAP和XML-RPC之类网络传输协议的一个简单方法;只要通信的两端都有Python(被 pickle的对象是用Python专有的格式表达的,而不是用XML文本)。

19.4.2. Picking in Action

19.4.2. Pickle实战

In more typical use, to pickle an object to a flat file, we just open the file in write mode and call the dump function:



% python
>>> table = {'a': [1, 2, 3],
'b': ['spam', 'eggs'],
'c': {'name':'bob'}}
>>> import pickle
>>> mydb = open('dbase', 'w')
>>> pickle.dump(table, mydb)

Notice the nesting in the object pickled herethe pickler handles arbitrary structures. To unpickle later in another session or program run, simply reopen the file and call load:



% python
>>> import pickle
>>> mydb = open('dbase', 'r')
>>> table = pickle.load(mydb)
>>> table
{'b': ['spam', 'eggs'], 'a': [1, 2, 3], 'c': {'name': 'bob'}}

The object you get back from unpickling has the same value and reference structure as the original, but it is located at a different address in memory. This is true whether the object is unpickled in the same or a future process. In Python-speak, the unpickled object is == but is not is:



% python
>>> import pickle
>>> f = open('temp', 'w')
>>> x = ['Hello', ('pickle', 'world')] # list with nested tuple
>>> pickle.dump(x, f)
>>> f.close( ) # close to flush changes
>>> f = open('temp', 'r')
>>> y = pickle.load(f)
>>> y
['Hello', ('pickle', 'world')]
>>> x == y, x is y
(True, False)

To make this process simpler still, the module in Example 19-1 wraps pickling and unpickling calls in functions that also open the files where the serialized form of the object is stored.


Example 19-1. PP3E\Dbase\filepickle.py

   1 import pickle
3 def saveDbase(filename, object):
4 file = open(filename, 'w')
5 pickle.dump(object, file) # pickle to file
6 file.close( ) # any file-like object will do
8 def loadDbase(filename):
9 file = open(filename, 'r')
10 object = pickle.load(file) # unpickle from file
11 file.close( ) # re-creates object in memory
12 return object

To store and fetch now, simply call these module functions; here they are in action managing a fairly complex structure with multiple references to the same nested objectthe nested list called L at first is stored only once in the file:



>>> from filepickle import *
>>> L = [0]
>>> D = {'x':0, 'y':L}
>>> table = {'A':L, 'B':D} # L appears twice
>>> saveDbase('myfile', table) # serialize to file

>>> from filepickle import *
>>> table = loadDbase('myfile') # reload/unpickle
>>> table
{'B': {'x': 0, 'y': [0]}, 'A': [0]}
>>> table['A'][0] = 1 # change shared object
>>> saveDbase('myfile', table) # rewrite to the file

>>> from filepickle import *
>>> print loadDbase('myfile') # both L's updated as expected
{'B': {'x': 0, 'y': [1]}, 'A': [1]}

Besides built-in types like the lists, tuples, and dictionaries of the examples so far, class instances may also be pickled to file-like objects. This provides a natural way to associate behavior with stored data (class methods process instance attributes) and provides a simple migration path (class changes made in module files are automatically picked up by stored instances). Here's a brief interactive demonstration:

除 了内置的类型,如以上例子中的列表,元组和字典,类实例也可被pickle到类似文件的对象中。这提供了一个自然的方式来关联行为与存储的数据(类方法处 理实例的属性),并提供了一条简单的迁移路径(被存储的实例将自动获得模块文件中对类的更改)。以下是个简短的交互演示:


>>> class Rec:
def _ _init_ _(self, hours):
self.hours = hours
def pay(self, rate=50):
return self.hours * rate

>>> bob = Rec(40)
>>> import pickle
>>> pickle.dump(bob, open('bobrec', 'w'))
>>> rec = pickle.load(open('bobrec'))
>>> rec.hours
>>> rec.pay( )

We'll explore how this works in more detail in conjunction with shelves later in this chapteras we'll see, although the pickle module can be used directly, it is also the underlying translation engine in both shelves and ZODB databases.


In fact, Python can pickle just about anything, except for:


Compiled code objects; functions and classes record just their names in pickles, to allow for later reimport and automatic acquisition of changes made in module files.


Instances of classes that do not follow class importability rules (more on this at the end of the section "Shelve Files," later in this chapter).

不遵守类可导入规则(class importability rule)的类实例(本章后面“Shelve文件”一节的尾部有更多相关内容)。

Instances of some built-in and user-defined types that are coded in C or depend upon transient operating system states (e.g., open file objects cannot be pickled).


A PicklingError is raised if an object cannot be pickled.


19.4.3. Pickler Protocols and cPickle

19.4.3. Pickler协议和cPickle

In recent Python releases, the pickler introduced the notion of protocolsstorage formats for pickled data. Specify the desired protocol by passing an extra parameter to the pickling calls (but not to unpickling calls: the protocol is automatically determined from the pickled data):



pickle.dump(object, file, protocol)

Pickled data may be created in either text or binary protocols. By default, the storage protocol is text (also known as protocol 0). In text mode, the files used to store pickled objects may be opened in text mode as in the earlier examples, and the pickled data is printable ASCII text, which can be read (it's essentially instructions for a stack machine).

Pickle 数据可以按文本协议或二进制协议产生。默认情况下,存储协议是文本协议(也称为0号协议)。在文本模式下,用来存储pickle对象的文件可以用文本模式 打开,如上述的例子,并且pickle的数据是可打印的ASCII文本,并且是可读的(这基本上是对堆栈机实现的指示)。

The alternative protocols (protocols 1 and 2) store the pickled data in binary format and require that files be opened in binary mode (e.g., rb, wb). Protocol 1 is the original binary format; protocol 2, added in Python 2.3, has improved support for pickling of new-style classes. Binary format is slightly more efficient, but it cannot be inspected. An older option to pickling calls, the bin argument, has been subsumed by using a pickling protocol higher than 0. The pickle module also provides a HIGHEST_PROTOCOL variable that can be passed in to automatically select the maximum value.

其 他协议(1号和2号协议 )以二进制格式存储pickle数据,并要求文件以二进制模式打开(例如:rb、wb)。1号协议是原始二进制格式;2号协议是Python 2.3增加的,它改善了对新型类pickle的支持。二进制格式效率更高一点,但它无法进行查看。旧的pickle调用有一个选项,即bin参数,现已被 归入使用大于0的协议。pickle模块还提供了一个HIGHEST_PROTOCOL变量,传入它可以自动选择最大的协议值。

One note: if you use the default text protocol, make sure you open pickle files in text mode later. On some platforms, opening text data in binary mode may cause unpickling errors due to line-end formats on Windows:



>>> f = open('temp', 'w')                  # text mode file on Windows
>>> pickle.dump(('ex', 'parrot'), f) # use default text protocol
>>> f.close( )
>>> pickle.load(open('temp', 'r')) # OK in text mode
('ex', 'parrot')
>>> pickle.load(open('temp', 'rb')) # fails in binary
Traceback (most recent call last):
File "<pyshell#337>", line 1, in -toplevel-
pickle.load(open('temp', 'rb'))
...lines deleted...
ValueError: insecure string pickle

One way to sidestep this potential issue is to always use binary mode for your files, even for the text pickle protocol. Since you must open files in binary mode for the binary pickler protocols anyhow (higher than the default 0), this isn't a bad habit to get into:



>>> f = open('temp', 'wb')                 # create in binary mode
>>> pickle.dump(('ex', 'parrot'), f) # use text protocol
>>> f.close( )
>>> pickle.load(open('temp', 'rb'))
('ex', 'parrot')
>>> pickle.load(open('temp', 'r'))
('ex', 'parrot')

Refer to Python's library manual for more information on the pickler. Also check out marshal, a module that serializes an object too, but can handle only simple object types. pickle is more general than marshal and is normally preferred.


And while you are flipping (or clicking) through that manual, be sure to also see the entries for the cPickle modulea reimplementation of pickle coded in C for faster performance. You can explicitly import cPickle for a substantial speed boost; its chief limitation is that you cannot subclass its versions of Pickle and Unpickle because they are functions, not classes (this is not required by most programs). The pickle and cPickle modules use compatible data formats, so they may be used interchangeably.

而 当你翻看(或点击)Python手册时,请一定也要看看cPickle模块的条目,它是pickle的C语言实现,性能上更快。您可以显式导入 cPickle替代pickle,以大幅提升速度;其主要的限制是,你不能继承该版本的Pickle和Unpickle,因为它们是函数,而不是类(多数 程序并不要求它们是类)。pickle和cPickle模块使用兼容的数据格式,所以它们可以互换使用。

If it is available in your Python, the shelve module automatically chooses the cPickle module for faster serialization, instead of pickle. I haven't explained shelve yet, but I will now.


posted on 2009-07-03 16:37 金庆 阅读(4871) 评论(1)  编辑 收藏 引用 所属分类: 6. Python


# re: (Python编程)Pickle对象 2009-09-03 15:25 pan
cool !  回复  更多评论

【推荐】超50万行VC++源码: 大型组态工控、电力仿真CAD与GIS源码库
网站导航: 博客园   IT新闻   BlogJava   知识库   博问   管理