mirror of https://github.com/python/cpython.git
886 lines
30 KiB
Plaintext
886 lines
30 KiB
Plaintext
|
=============================================================================
|
|||
|
Python Unicode Integration Proposal Version: 1.2
|
|||
|
-----------------------------------------------------------------------------
|
|||
|
|
|||
|
|
|||
|
Introduction:
|
|||
|
-------------
|
|||
|
|
|||
|
The idea of this proposal is to add native Unicode 3.0 support to
|
|||
|
Python in a way that makes use of Unicode strings as simple as
|
|||
|
possible without introducing too many pitfalls along the way.
|
|||
|
|
|||
|
Since this goal is not easy to achieve -- strings being one of the
|
|||
|
most fundamental objects in Python --, we expect this proposal to
|
|||
|
undergo some significant refinements.
|
|||
|
|
|||
|
Note that the current version of this proposal is still a bit unsorted
|
|||
|
due to the many different aspects of the Unicode-Python integration.
|
|||
|
|
|||
|
The latest version of this document is always available at:
|
|||
|
|
|||
|
http://starship.skyport.net/~lemburg/unicode-proposal.txt
|
|||
|
|
|||
|
Older versions are available as:
|
|||
|
|
|||
|
http://starship.skyport.net/~lemburg/unicode-proposal-X.X.txt
|
|||
|
|
|||
|
|
|||
|
Conventions:
|
|||
|
------------
|
|||
|
|
|||
|
<EFBFBD> In examples we use u = Unicode object and s = Python string
|
|||
|
|
|||
|
<EFBFBD> 'XXX' markings indicate points of discussion (PODs)
|
|||
|
|
|||
|
|
|||
|
General Remarks:
|
|||
|
----------------
|
|||
|
|
|||
|
<EFBFBD> Unicode encoding names should be lower case on output and
|
|||
|
case-insensitive on input (they will be converted to lower case
|
|||
|
by all APIs taking an encoding name as input).
|
|||
|
|
|||
|
Encoding names should follow the name conventions as used by the
|
|||
|
Unicode Consortium: spaces are converted to hyphens, e.g. 'utf 16' is
|
|||
|
written as 'utf-16'.
|
|||
|
|
|||
|
Codec modules should use the same names, but with hyphens converted
|
|||
|
to underscores, e.g. utf_8, utf_16, iso_8859_1.
|
|||
|
|
|||
|
<EFBFBD> The <default encoding> should be the widely used 'utf-8' format. This
|
|||
|
is very close to the standard 7-bit ASCII format and thus resembles the
|
|||
|
standard used programming nowadays in most aspects.
|
|||
|
|
|||
|
|
|||
|
Unicode Constructors:
|
|||
|
---------------------
|
|||
|
|
|||
|
Python should provide a built-in constructor for Unicode strings which
|
|||
|
is available through __builtins__:
|
|||
|
|
|||
|
u = unicode(encoded_string[,encoding=<default encoding>][,errors="strict"])
|
|||
|
|
|||
|
u = u'<unicode-escape encoded Python string>'
|
|||
|
|
|||
|
u = ur'<raw-unicode-escape encoded Python string>'
|
|||
|
|
|||
|
With the 'unicode-escape' encoding being defined as:
|
|||
|
|
|||
|
<EFBFBD> all non-escape characters represent themselves as Unicode ordinal
|
|||
|
(e.g. 'a' -> U+0061).
|
|||
|
|
|||
|
<EFBFBD> all existing defined Python escape sequences are interpreted as
|
|||
|
Unicode ordinals; note that \xXXXX can represent all Unicode
|
|||
|
ordinals, and \OOO (octal) can represent Unicode ordinals up to U+01FF.
|
|||
|
|
|||
|
<EFBFBD> a new escape sequence, \uXXXX, represents U+XXXX; it is a syntax
|
|||
|
error to have fewer than 4 digits after \u.
|
|||
|
|
|||
|
For an explanation of possible values for errors see the Codec section
|
|||
|
below.
|
|||
|
|
|||
|
Examples:
|
|||
|
|
|||
|
u'abc' -> U+0061 U+0062 U+0063
|
|||
|
u'\u1234' -> U+1234
|
|||
|
u'abc\u1234\n' -> U+0061 U+0062 U+0063 U+1234 U+005c
|
|||
|
|
|||
|
The 'raw-unicode-escape' encoding is defined as follows:
|
|||
|
|
|||
|
<EFBFBD> \uXXXX sequence represent the U+XXXX Unicode character if and
|
|||
|
only if the number of leading backslashes is odd
|
|||
|
|
|||
|
<EFBFBD> all other characters represent themselves as Unicode ordinal
|
|||
|
(e.g. 'b' -> U+0062)
|
|||
|
|
|||
|
|
|||
|
Note that you should provide some hint to the encoding you used to
|
|||
|
write your programs as pragma line in one the first few comment lines
|
|||
|
of the source file (e.g. '# source file encoding: latin-1'). If you
|
|||
|
only use 7-bit ASCII then everything is fine and no such notice is
|
|||
|
needed, but if you include Latin-1 characters not defined in ASCII, it
|
|||
|
may well be worthwhile including a hint since people in other
|
|||
|
countries will want to be able to read you source strings too.
|
|||
|
|
|||
|
|
|||
|
Unicode Type Object:
|
|||
|
--------------------
|
|||
|
|
|||
|
Unicode objects should have the type UnicodeType with type name
|
|||
|
'unicode', made available through the standard types module.
|
|||
|
|
|||
|
|
|||
|
Unicode Output:
|
|||
|
---------------
|
|||
|
|
|||
|
Unicode objects have a method .encode([encoding=<default encoding>])
|
|||
|
which returns a Python string encoding the Unicode string using the
|
|||
|
given scheme (see Codecs).
|
|||
|
|
|||
|
print u := print u.encode() # using the <default encoding>
|
|||
|
|
|||
|
str(u) := u.encode() # using the <default encoding>
|
|||
|
|
|||
|
repr(u) := "u%s" % repr(u.encode('unicode-escape'))
|
|||
|
|
|||
|
Also see Internal Argument Parsing and Buffer Interface for details on
|
|||
|
how other APIs written in C will treat Unicode objects.
|
|||
|
|
|||
|
|
|||
|
Unicode Ordinals:
|
|||
|
-----------------
|
|||
|
|
|||
|
Since Unicode 3.0 has a 32-bit ordinal character set, the implementation
|
|||
|
should provide 32-bit aware ordinal conversion APIs:
|
|||
|
|
|||
|
ord(u[:1]) (this is the standard ord() extended to work with Unicode
|
|||
|
objects)
|
|||
|
--> Unicode ordinal number (32-bit)
|
|||
|
|
|||
|
unichr(i)
|
|||
|
--> Unicode object for character i (provided it is 32-bit);
|
|||
|
ValueError otherwise
|
|||
|
|
|||
|
Both APIs should go into __builtins__ just like their string
|
|||
|
counterparts ord() and chr().
|
|||
|
|
|||
|
Note that Unicode provides space for private encodings. Usage of these
|
|||
|
can cause different output representations on different machines. This
|
|||
|
problem is not a Python or Unicode problem, but a machine setup and
|
|||
|
maintenance one.
|
|||
|
|
|||
|
|
|||
|
Comparison & Hash Value:
|
|||
|
------------------------
|
|||
|
|
|||
|
Unicode objects should compare equal to other objects after these
|
|||
|
other objects have been coerced to Unicode. For strings this means
|
|||
|
that they are interpreted as Unicode string using the <default
|
|||
|
encoding>.
|
|||
|
|
|||
|
For the same reason, Unicode objects should return the same hash value
|
|||
|
as their UTF-8 equivalent strings.
|
|||
|
|
|||
|
Coercion:
|
|||
|
---------
|
|||
|
|
|||
|
Using Python strings and Unicode objects to form new objects should
|
|||
|
always coerce to the more precise format, i.e. Unicode objects.
|
|||
|
|
|||
|
u + s := u + unicode(s)
|
|||
|
|
|||
|
s + u := unicode(s) + u
|
|||
|
|
|||
|
All string methods should delegate the call to an equivalent Unicode
|
|||
|
object method call by converting all envolved strings to Unicode and
|
|||
|
then applying the arguments to the Unicode method of the same name,
|
|||
|
e.g.
|
|||
|
|
|||
|
string.join((s,u),sep) := (s + sep) + u
|
|||
|
|
|||
|
sep.join((s,u)) := (s + sep) + u
|
|||
|
|
|||
|
For a discussion of %-formatting w/r to Unicode objects, see
|
|||
|
Formatting Markers.
|
|||
|
|
|||
|
|
|||
|
Exceptions:
|
|||
|
-----------
|
|||
|
|
|||
|
UnicodeError is defined in the exceptions module as subclass of
|
|||
|
ValueError. It is available at the C level via PyExc_UnicodeError.
|
|||
|
All exceptions related to Unicode encoding/decoding should be
|
|||
|
subclasses of UnicodeError.
|
|||
|
|
|||
|
|
|||
|
Codecs (Coder/Decoders) Lookup:
|
|||
|
-------------------------------
|
|||
|
|
|||
|
A Codec (see Codec Interface Definition) search registry should be
|
|||
|
implemented by a module "codecs":
|
|||
|
|
|||
|
codecs.register(search_function)
|
|||
|
|
|||
|
Search functions are expected to take one argument, the encoding name
|
|||
|
in all lower case letters, and return a tuple of functions (encoder,
|
|||
|
decoder, stream_reader, stream_writer) taking the following arguments:
|
|||
|
|
|||
|
encoder and decoder:
|
|||
|
These must be functions or methods which have the same
|
|||
|
interface as the .encode/.decode methods of Codec instances
|
|||
|
(see Codec Interface). The functions/methods are expected to
|
|||
|
work in a stateless mode.
|
|||
|
|
|||
|
stream_reader and stream_writer:
|
|||
|
These need to be factory functions with the following
|
|||
|
interface:
|
|||
|
|
|||
|
factory(stream,errors='strict')
|
|||
|
|
|||
|
The factory functions must return objects providing
|
|||
|
the interfaces defined by StreamWriter/StreamReader resp.
|
|||
|
(see Codec Interface). Stream codecs can maintain state.
|
|||
|
|
|||
|
Possible values for errors are defined in the Codec
|
|||
|
section below.
|
|||
|
|
|||
|
In case a search function cannot find a given encoding, it should
|
|||
|
return None.
|
|||
|
|
|||
|
Aliasing support for encodings is left to the search functions
|
|||
|
to implement.
|
|||
|
|
|||
|
The codecs module will maintain an encoding cache for performance
|
|||
|
reasons. Encodings are first looked up in the cache. If not found, the
|
|||
|
list of registered search functions is scanned. If no codecs tuple is
|
|||
|
found, a LookupError is raised. Otherwise, the codecs tuple is stored
|
|||
|
in the cache and returned to the caller.
|
|||
|
|
|||
|
To query the Codec instance the following API should be used:
|
|||
|
|
|||
|
codecs.lookup(encoding)
|
|||
|
|
|||
|
This will either return the found codecs tuple or raise a LookupError.
|
|||
|
|
|||
|
|
|||
|
Standard Codecs:
|
|||
|
----------------
|
|||
|
|
|||
|
Standard codecs should live inside an encodings/ package directory in the
|
|||
|
Standard Python Code Library. The __init__.py file of that directory should
|
|||
|
include a Codec Lookup compatible search function implementing a lazy module
|
|||
|
based codec lookup.
|
|||
|
|
|||
|
Python should provide a few standard codecs for the most relevant
|
|||
|
encodings, e.g.
|
|||
|
|
|||
|
'utf-8': 8-bit variable length encoding
|
|||
|
'utf-16': 16-bit variable length encoding (litte/big endian)
|
|||
|
'utf-16-le': utf-16 but explicitly little endian
|
|||
|
'utf-16-be': utf-16 but explicitly big endian
|
|||
|
'ascii': 7-bit ASCII codepage
|
|||
|
'iso-8859-1': ISO 8859-1 (Latin 1) codepage
|
|||
|
'unicode-escape': See Unicode Constructors for a definition
|
|||
|
'raw-unicode-escape': See Unicode Constructors for a definition
|
|||
|
'native': Dump of the Internal Format used by Python
|
|||
|
|
|||
|
Common aliases should also be provided per default, e.g. 'latin-1'
|
|||
|
for 'iso-8859-1'.
|
|||
|
|
|||
|
Note: 'utf-16' should be implemented by using and requiring byte order
|
|||
|
marks (BOM) for file input/output.
|
|||
|
|
|||
|
All other encodings such as the CJK ones to support Asian scripts
|
|||
|
should be implemented in seperate packages which do not get included
|
|||
|
in the core Python distribution and are not a part of this proposal.
|
|||
|
|
|||
|
|
|||
|
Codecs Interface Definition:
|
|||
|
----------------------------
|
|||
|
|
|||
|
The following base class should be defined in the module
|
|||
|
"codecs". They provide not only templates for use by encoding module
|
|||
|
implementors, but also define the interface which is expected by the
|
|||
|
Unicode implementation.
|
|||
|
|
|||
|
Note that the Codec Interface defined here is well suitable for a
|
|||
|
larger range of applications. The Unicode implementation expects
|
|||
|
Unicode objects on input for .encode() and .write() and character
|
|||
|
buffer compatible objects on input for .decode(). Output of .encode()
|
|||
|
and .read() should be a Python string and .decode() must return an
|
|||
|
Unicode object.
|
|||
|
|
|||
|
First, we have the stateless encoders/decoders. These do not work in
|
|||
|
chunks as the stream codecs (see below) do, because all components are
|
|||
|
expected to be available in memory.
|
|||
|
|
|||
|
class Codec:
|
|||
|
|
|||
|
""" Defines the interface for stateless encoders/decoders.
|
|||
|
|
|||
|
The .encode()/.decode() methods may implement different error
|
|||
|
handling schemes by providing the errors argument. These
|
|||
|
string values are defined:
|
|||
|
|
|||
|
'strict' - raise an error (or a subclass)
|
|||
|
'ignore' - ignore the character and continue with the next
|
|||
|
'replace' - replace with a suitable replacement character;
|
|||
|
Python will use the official U+FFFD REPLACEMENT
|
|||
|
CHARACTER for the builtin Unicode codecs.
|
|||
|
|
|||
|
"""
|
|||
|
def encode(self,input,errors='strict'):
|
|||
|
|
|||
|
""" Encodes the object intput and returns a tuple (output
|
|||
|
object, length consumed).
|
|||
|
|
|||
|
errors defines the error handling to apply. It defaults to
|
|||
|
'strict' handling.
|
|||
|
|
|||
|
The method may not store state in the Codec instance. Use
|
|||
|
SteamCodec for codecs which have to keep state in order to
|
|||
|
make encoding/decoding efficient.
|
|||
|
|
|||
|
"""
|
|||
|
...
|
|||
|
|
|||
|
def decode(self,input,errors='strict'):
|
|||
|
|
|||
|
""" Decodes the object input and returns a tuple (output
|
|||
|
object, length consumed).
|
|||
|
|
|||
|
input must be an object which provides the bf_getreadbuf
|
|||
|
buffer slot. Python strings, buffer objects and memory
|
|||
|
mapped files are examples of objects providing this slot.
|
|||
|
|
|||
|
errors defines the error handling to apply. It defaults to
|
|||
|
'strict' handling.
|
|||
|
|
|||
|
The method may not store state in the Codec instance. Use
|
|||
|
SteamCodec for codecs which have to keep state in order to
|
|||
|
make encoding/decoding efficient.
|
|||
|
|
|||
|
"""
|
|||
|
...
|
|||
|
|
|||
|
StreamWriter and StreamReader define the interface for stateful
|
|||
|
encoders/decoders which work on streams. These allow processing of the
|
|||
|
data in chunks to efficiently use memory. If you have large strings in
|
|||
|
memory, you may want to wrap them with cStringIO objects and then use
|
|||
|
these codecs on them to be able to do chunk processing as well,
|
|||
|
e.g. to provide progress information to the user.
|
|||
|
|
|||
|
class StreamWriter(Codec):
|
|||
|
|
|||
|
def __init__(self,stream,errors='strict'):
|
|||
|
|
|||
|
""" Creates a StreamWriter instance.
|
|||
|
|
|||
|
stream must be a file-like object open for writing
|
|||
|
(binary) data.
|
|||
|
|
|||
|
The StreamWriter may implement different error handling
|
|||
|
schemes by providing the errors keyword argument. These
|
|||
|
parameters are defined:
|
|||
|
|
|||
|
'strict' - raise a ValueError (or a subclass)
|
|||
|
'ignore' - ignore the character and continue with the next
|
|||
|
'replace'- replace with a suitable replacement character
|
|||
|
|
|||
|
"""
|
|||
|
self.stream = stream
|
|||
|
self.errors = errors
|
|||
|
|
|||
|
def write(self,object):
|
|||
|
|
|||
|
""" Writes the object's contents encoded to self.stream.
|
|||
|
"""
|
|||
|
data, consumed = self.encode(object,self.errors)
|
|||
|
self.stream.write(data)
|
|||
|
|
|||
|
def reset(self):
|
|||
|
|
|||
|
""" Flushes and resets the codec buffers used for keeping state.
|
|||
|
|
|||
|
Calling this method should ensure that the data on the
|
|||
|
output is put into a clean state, that allows appending
|
|||
|
of new fresh data without having to rescan the whole
|
|||
|
stream to recover state.
|
|||
|
|
|||
|
"""
|
|||
|
pass
|
|||
|
|
|||
|
def __getattr__(self,name,
|
|||
|
|
|||
|
getattr=getattr):
|
|||
|
|
|||
|
""" Inherit all other methods from the underlying stream.
|
|||
|
"""
|
|||
|
return getattr(self.stream,name)
|
|||
|
|
|||
|
class StreamReader(Codec):
|
|||
|
|
|||
|
def __init__(self,stream,errors='strict'):
|
|||
|
|
|||
|
""" Creates a StreamReader instance.
|
|||
|
|
|||
|
stream must be a file-like object open for reading
|
|||
|
(binary) data.
|
|||
|
|
|||
|
The StreamReader may implement different error handling
|
|||
|
schemes by providing the errors keyword argument. These
|
|||
|
parameters are defined:
|
|||
|
|
|||
|
'strict' - raise a ValueError (or a subclass)
|
|||
|
'ignore' - ignore the character and continue with the next
|
|||
|
'replace'- replace with a suitable replacement character;
|
|||
|
|
|||
|
"""
|
|||
|
self.stream = stream
|
|||
|
self.errors = errors
|
|||
|
|
|||
|
def read(self,size=-1):
|
|||
|
|
|||
|
""" Decodes data from the stream self.stream and returns the
|
|||
|
resulting object.
|
|||
|
|
|||
|
size indicates the approximate maximum number of bytes to
|
|||
|
read from the stream for decoding purposes. The decoder
|
|||
|
can modify this setting as appropriate. The default value
|
|||
|
-1 indicates to read and decode as much as possible. size
|
|||
|
is intended to prevent having to decode huge files in one
|
|||
|
step.
|
|||
|
|
|||
|
The method should use a greedy read strategy meaning that
|
|||
|
it should read as much data as is allowed within the
|
|||
|
definition of the encoding and the given size, e.g. if
|
|||
|
optional encoding endings or state markers are available
|
|||
|
on the stream, these should be read too.
|
|||
|
|
|||
|
"""
|
|||
|
# Unsliced reading:
|
|||
|
if size < 0:
|
|||
|
return self.decode(self.stream.read())[0]
|
|||
|
|
|||
|
# Sliced reading:
|
|||
|
read = self.stream.read
|
|||
|
decode = self.decode
|
|||
|
data = read(size)
|
|||
|
i = 0
|
|||
|
while 1:
|
|||
|
try:
|
|||
|
object, decodedbytes = decode(data)
|
|||
|
except ValueError,why:
|
|||
|
# This method is slow but should work under pretty much
|
|||
|
# all conditions; at most 10 tries are made
|
|||
|
i = i + 1
|
|||
|
newdata = read(1)
|
|||
|
if not newdata or i > 10:
|
|||
|
raise
|
|||
|
data = data + newdata
|
|||
|
else:
|
|||
|
return object
|
|||
|
|
|||
|
def reset(self):
|
|||
|
|
|||
|
""" Resets the codec buffers used for keeping state.
|
|||
|
|
|||
|
Note that no stream repositioning should take place.
|
|||
|
This method is primarely intended to be able to recover
|
|||
|
from decoding errors.
|
|||
|
|
|||
|
"""
|
|||
|
pass
|
|||
|
|
|||
|
def __getattr__(self,name,
|
|||
|
|
|||
|
getattr=getattr):
|
|||
|
|
|||
|
""" Inherit all other methods from the underlying stream.
|
|||
|
"""
|
|||
|
return getattr(self.stream,name)
|
|||
|
|
|||
|
XXX What about .readline(), .readlines() ? These could be implemented
|
|||
|
using .read() as generic functions instead of requiring their
|
|||
|
implementation by all codecs. Also see Line Breaks.
|
|||
|
|
|||
|
Stream codec implementors are free to combine the StreamWriter and
|
|||
|
StreamReader interfaces into one class. Even combining all these with
|
|||
|
the Codec class should be possible.
|
|||
|
|
|||
|
Implementors are free to add additional methods to enhance the codec
|
|||
|
functionality or provide extra state information needed for them to
|
|||
|
work. The internal codec implementation will only use the above
|
|||
|
interfaces, though.
|
|||
|
|
|||
|
It is not required by the Unicode implementation to use these base
|
|||
|
classes, only the interfaces must match; this allows writing Codecs as
|
|||
|
extensions types.
|
|||
|
|
|||
|
As guideline, large mapping tables should be implemented using static
|
|||
|
C data in separate (shared) extension modules. That way multiple
|
|||
|
processes can share the same data.
|
|||
|
|
|||
|
A tool to auto-convert Unicode mapping files to mapping modules should be
|
|||
|
provided to simplify support for additional mappings (see References).
|
|||
|
|
|||
|
|
|||
|
Whitespace:
|
|||
|
-----------
|
|||
|
|
|||
|
The .split() method will have to know about what is considered
|
|||
|
whitespace in Unicode.
|
|||
|
|
|||
|
|
|||
|
Case Conversion:
|
|||
|
----------------
|
|||
|
|
|||
|
Case conversion is rather complicated with Unicode data, since there
|
|||
|
are many different conditions to respect. See
|
|||
|
|
|||
|
http://www.unicode.org/unicode/reports/tr13/
|
|||
|
|
|||
|
for some guidelines on implementing case conversion.
|
|||
|
|
|||
|
For Python, we should only implement the 1-1 conversions included in
|
|||
|
Unicode. Locale dependent and other special case conversions (see the
|
|||
|
Unicode standard file SpecialCasing.txt) should be left to user land
|
|||
|
routines and not go into the core interpreter.
|
|||
|
|
|||
|
The methods .capitalize() and .iscapitalized() should follow the case
|
|||
|
mapping algorithm defined in the above technical report as closely as
|
|||
|
possible.
|
|||
|
|
|||
|
|
|||
|
Line Breaks:
|
|||
|
------------
|
|||
|
|
|||
|
Line breaking should be done for all Unicode characters having the B
|
|||
|
property as well as the combinations CRLF, CR, LF (interpreted in that
|
|||
|
order) and other special line separators defined by the standard.
|
|||
|
|
|||
|
The Unicode type should provide a .splitlines() method which returns a
|
|||
|
list of lines according to the above specification. See Unicode
|
|||
|
Methods.
|
|||
|
|
|||
|
|
|||
|
Unicode Character Properties:
|
|||
|
-----------------------------
|
|||
|
|
|||
|
A separate module "unicodedata" should provide a compact interface to
|
|||
|
all Unicode character properties defined in the standard's
|
|||
|
UnicodeData.txt file.
|
|||
|
|
|||
|
Among other things, these properties provide ways to recognize
|
|||
|
numbers, digits, spaces, whitespace, etc.
|
|||
|
|
|||
|
Since this module will have to provide access to all Unicode
|
|||
|
characters, it will eventually have to contain the data from
|
|||
|
UnicodeData.txt which takes up around 600kB. For this reason, the data
|
|||
|
should be stored in static C data. This enables compilation as shared
|
|||
|
module which the underlying OS can shared between processes (unlike
|
|||
|
normal Python code modules).
|
|||
|
|
|||
|
There should be a standard Python interface for accessing this information
|
|||
|
so that other implementors can plug in their own possibly enhanced versions,
|
|||
|
e.g. ones that do decompressing of the data on-the-fly.
|
|||
|
|
|||
|
|
|||
|
Private Code Point Areas:
|
|||
|
-------------------------
|
|||
|
|
|||
|
Support for these is left to user land Codecs and not explicitly
|
|||
|
intergrated into the core. Note that due to the Internal Format being
|
|||
|
implemented, only the area between \uE000 and \uF8FF is useable for
|
|||
|
private encodings.
|
|||
|
|
|||
|
|
|||
|
Internal Format:
|
|||
|
----------------
|
|||
|
|
|||
|
The internal format for Unicode objects should use a Python specific
|
|||
|
fixed format <PythonUnicode> implemented as 'unsigned short' (or
|
|||
|
another unsigned numeric type having 16 bits). Byte order is platform
|
|||
|
dependent.
|
|||
|
|
|||
|
This format will hold UTF-16 encodings of the corresponding Unicode
|
|||
|
ordinals. The Python Unicode implementation will address these values
|
|||
|
as if they were UCS-2 values. UCS-2 and UTF-16 are the same for all
|
|||
|
currently defined Unicode character points. UTF-16 without surrogates
|
|||
|
provides access to about 64k characters and covers all characters in
|
|||
|
the Basic Multilingual Plane (BMP) of Unicode.
|
|||
|
|
|||
|
It is the Codec's responsibility to ensure that the data they pass to
|
|||
|
the Unicode object constructor repects this assumption. The
|
|||
|
constructor does not check the data for Unicode compliance or use of
|
|||
|
surrogates.
|
|||
|
|
|||
|
Future implementations can extend the 32 bit restriction to the full
|
|||
|
set of all UTF-16 addressable characters (around 1M characters).
|
|||
|
|
|||
|
The Unicode API should provide inteface routines from <PythonUnicode>
|
|||
|
to the compiler's wchar_t which can be 16 or 32 bit depending on the
|
|||
|
compiler/libc/platform being used.
|
|||
|
|
|||
|
Unicode objects should have a pointer to a cached Python string object
|
|||
|
<defencstr> holding the object's value using the current <default
|
|||
|
encoding>. This is needed for performance and internal parsing (see
|
|||
|
Internal Argument Parsing) reasons. The buffer is filled when the
|
|||
|
first conversion request to the <default encoding> is issued on the
|
|||
|
object.
|
|||
|
|
|||
|
Interning is not needed (for now), since Python identifiers are
|
|||
|
defined as being ASCII only.
|
|||
|
|
|||
|
codecs.BOM should return the byte order mark (BOM) for the format
|
|||
|
used internally. The codecs module should provide the following
|
|||
|
additional constants for convenience and reference (codecs.BOM will
|
|||
|
either be BOM_BE or BOM_LE depending on the platform):
|
|||
|
|
|||
|
BOM_BE: '\376\377'
|
|||
|
(corresponds to Unicode U+0000FEFF in UTF-16 on big endian
|
|||
|
platforms == ZERO WIDTH NO-BREAK SPACE)
|
|||
|
|
|||
|
BOM_LE: '\377\376'
|
|||
|
(corresponds to Unicode U+0000FFFE in UTF-16 on little endian
|
|||
|
platforms == defined as being an illegal Unicode character)
|
|||
|
|
|||
|
BOM4_BE: '\000\000\376\377'
|
|||
|
(corresponds to Unicode U+0000FEFF in UCS-4)
|
|||
|
|
|||
|
BOM4_LE: '\377\376\000\000'
|
|||
|
(corresponds to Unicode U+0000FFFE in UCS-4)
|
|||
|
|
|||
|
Note that Unicode sees big endian byte order as being "correct". The
|
|||
|
swapped order is taken to be an indicator for a "wrong" format, hence
|
|||
|
the illegal character definition.
|
|||
|
|
|||
|
The configure script should provide aid in deciding whether Python can
|
|||
|
use the native wchar_t type or not (it has to be a 16-bit unsigned
|
|||
|
type).
|
|||
|
|
|||
|
|
|||
|
Buffer Interface:
|
|||
|
-----------------
|
|||
|
|
|||
|
Implement the buffer interface using the <defencstr> Python string
|
|||
|
object as basis for bf_getcharbuf (corresponds to the "t#" argument
|
|||
|
parsing marker) and the internal buffer for bf_getreadbuf (corresponds
|
|||
|
to the "s#" argument parsing marker). If bf_getcharbuf is requested
|
|||
|
and the <defencstr> object does not yet exist, it is created first.
|
|||
|
|
|||
|
This has the advantage of being able to write to output streams (which
|
|||
|
typically use this interface) without additional specification of the
|
|||
|
encoding to use.
|
|||
|
|
|||
|
The internal format can also be accessed using the 'unicode-internal'
|
|||
|
codec, e.g. via u.encode('unicode-internal').
|
|||
|
|
|||
|
|
|||
|
Pickle/Marshalling:
|
|||
|
-------------------
|
|||
|
|
|||
|
Should have native Unicode object support. The objects should be
|
|||
|
encoded using platform independent encodings.
|
|||
|
|
|||
|
Marshal should use UTF-8 and Pickle should either choose
|
|||
|
Raw-Unicode-Escape (in text mode) or UTF-8 (in binary mode) as
|
|||
|
encoding. Using UTF-8 instead of UTF-16 has the advantage of
|
|||
|
eliminating the need to store a BOM mark.
|
|||
|
|
|||
|
|
|||
|
Regular Expressions:
|
|||
|
--------------------
|
|||
|
|
|||
|
Secret Labs AB is working on a Unicode-aware regular expression
|
|||
|
machinery. It works on plain 8-bit, UCS-2, and (optionally) UCS-4
|
|||
|
internal character buffers.
|
|||
|
|
|||
|
Also see
|
|||
|
|
|||
|
http://www.unicode.org/unicode/reports/tr18/
|
|||
|
|
|||
|
for some remarks on how to treat Unicode REs.
|
|||
|
|
|||
|
|
|||
|
Formatting Markers:
|
|||
|
-------------------
|
|||
|
|
|||
|
Format markers are used in Python format strings. If Python strings
|
|||
|
are used as format strings, the following interpretations should be in
|
|||
|
effect:
|
|||
|
|
|||
|
'%s': '%s' does str(u) for Unicode objects embedded
|
|||
|
in Python strings, so the output will be
|
|||
|
u.encode(<default encoding>)
|
|||
|
|
|||
|
In case the format string is an Unicode object, all parameters are coerced
|
|||
|
to Unicode first and then put together and formatted according to the format
|
|||
|
string. Numbers are first converted to strings and then to Unicode.
|
|||
|
|
|||
|
'%s': Python strings are interpreted as Unicode
|
|||
|
string using the <default encoding>. Unicode
|
|||
|
objects are taken as is.
|
|||
|
|
|||
|
All other string formatters should work accordingly.
|
|||
|
|
|||
|
Example:
|
|||
|
|
|||
|
u"%s %s" % (u"abc", "abc") == u"abc abc"
|
|||
|
|
|||
|
|
|||
|
Internal Argument Parsing:
|
|||
|
--------------------------
|
|||
|
|
|||
|
These markers are used by the PyArg_ParseTuple() APIs:
|
|||
|
|
|||
|
'U': Check for Unicode object and return a pointer to it
|
|||
|
|
|||
|
's': For Unicode objects: auto convert them to the <default encoding>
|
|||
|
and return a pointer to the object's <defencstr> buffer.
|
|||
|
|
|||
|
's#': Access to the Unicode object via the bf_getreadbuf buffer interface
|
|||
|
(see Buffer Interface); note that the length relates to the buffer
|
|||
|
length, not the Unicode string length (this may be different
|
|||
|
depending on the Internal Format).
|
|||
|
|
|||
|
't#': Access to the Unicode object via the bf_getcharbuf buffer interface
|
|||
|
(see Buffer Interface); note that the length relates to the buffer
|
|||
|
length, not necessarily to the Unicode string length (this may
|
|||
|
be different depending on the <default encoding>).
|
|||
|
|
|||
|
|
|||
|
File/Stream Output:
|
|||
|
-------------------
|
|||
|
|
|||
|
Since file.write(object) and most other stream writers use the "s#"
|
|||
|
argument parsing marker for binary files and "t#" for text files, the
|
|||
|
buffer interface implementation determines the encoding to use (see
|
|||
|
Buffer Interface).
|
|||
|
|
|||
|
For explicit handling of files using Unicode, the standard
|
|||
|
stream codecs as available through the codecs module should
|
|||
|
be used.
|
|||
|
|
|||
|
XXX There should be a short-cut open(filename,mode,encoding) available which
|
|||
|
also assures that mode contains the 'b' character when needed.
|
|||
|
|
|||
|
|
|||
|
File/Stream Input:
|
|||
|
------------------
|
|||
|
|
|||
|
Only the user knows what encoding the input data uses, so no special
|
|||
|
magic is applied. The user will have to explicitly convert the string
|
|||
|
data to Unicode objects as needed or use the file wrappers defined in
|
|||
|
the codecs module (see File/Stream Output).
|
|||
|
|
|||
|
|
|||
|
Unicode Methods & Attributes:
|
|||
|
-----------------------------
|
|||
|
|
|||
|
All Python string methods, plus:
|
|||
|
|
|||
|
.encode([encoding=<default encoding>][,errors="strict"])
|
|||
|
--> see Unicode Output
|
|||
|
|
|||
|
.splitlines([include_breaks=0])
|
|||
|
--> breaks the Unicode string into a list of (Unicode) lines;
|
|||
|
returns the lines with line breaks included, if include_breaks
|
|||
|
is true. See Line Breaks for a specification of how line breaking
|
|||
|
is done.
|
|||
|
|
|||
|
|
|||
|
Code Base:
|
|||
|
----------
|
|||
|
|
|||
|
We should use Fredrik Lundh's Unicode object implementation as basis.
|
|||
|
It already implements most of the string methods needed and provides a
|
|||
|
well written code base which we can build upon.
|
|||
|
|
|||
|
The object sharing implemented in Fredrik's implementation should
|
|||
|
be dropped.
|
|||
|
|
|||
|
|
|||
|
Test Cases:
|
|||
|
-----------
|
|||
|
|
|||
|
Test cases should follow those in Lib/test/test_string.py and include
|
|||
|
additional checks for the Codec Registry and the Standard Codecs.
|
|||
|
|
|||
|
|
|||
|
References:
|
|||
|
-----------
|
|||
|
|
|||
|
Unicode Consortium:
|
|||
|
http://www.unicode.org/
|
|||
|
|
|||
|
Unicode FAQ:
|
|||
|
http://www.unicode.org/unicode/faq/
|
|||
|
|
|||
|
Unicode 3.0:
|
|||
|
http://www.unicode.org/unicode/standard/versions/Unicode3.0.html
|
|||
|
|
|||
|
Unicode-TechReports:
|
|||
|
http://www.unicode.org/unicode/reports/techreports.html
|
|||
|
|
|||
|
Unicode-Mappings:
|
|||
|
ftp://ftp.unicode.org/Public/MAPPINGS/
|
|||
|
|
|||
|
Introduction to Unicode (a little outdated by still nice to read):
|
|||
|
http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html
|
|||
|
|
|||
|
Encodings:
|
|||
|
|
|||
|
Overview:
|
|||
|
http://czyborra.com/utf/
|
|||
|
|
|||
|
UTC-2:
|
|||
|
http://www.uazone.com/multiling/unicode/ucs2.html
|
|||
|
|
|||
|
UTF-7:
|
|||
|
Defined in RFC2152, e.g.
|
|||
|
http://www.uazone.com/multiling/ml-docs/rfc2152.txt
|
|||
|
|
|||
|
UTF-8:
|
|||
|
Defined in RFC2279, e.g.
|
|||
|
http://info.internet.isi.edu/in-notes/rfc/files/rfc2279.txt
|
|||
|
|
|||
|
UTF-16:
|
|||
|
http://www.uazone.com/multiling/unicode/wg2n1035.html
|
|||
|
|
|||
|
|
|||
|
History of this Proposal:
|
|||
|
-------------------------
|
|||
|
1.2:
|
|||
|
1.1: Added note about comparisons and hash values. Added note about
|
|||
|
case mapping algorithms. Changed stream codecs .read() and
|
|||
|
.write() method to match the standard file-like object methods
|
|||
|
(bytes consumed information is no longer returned by the methods)
|
|||
|
1.0: changed encode Codec method to be symmetric to the decode method
|
|||
|
(they both return (object, data consumed) now and thus become
|
|||
|
interchangeable); removed __init__ method of Codec class (the
|
|||
|
methods are stateless) and moved the errors argument down to the
|
|||
|
methods; made the Codec design more generic w/r to type of input
|
|||
|
and output objects; changed StreamWriter.flush to StreamWriter.reset
|
|||
|
in order to avoid overriding the stream's .flush() method;
|
|||
|
renamed .breaklines() to .splitlines(); renamed the module unicodec
|
|||
|
to codecs; modified the File I/O section to refer to the stream codecs.
|
|||
|
0.9: changed errors keyword argument definition; added 'replace' error
|
|||
|
handling; changed the codec APIs to accept buffer like objects on
|
|||
|
input; some minor typo fixes; added Whitespace section and
|
|||
|
included references for Unicode characters that have the whitespace
|
|||
|
and the line break characteristic; added note that search functions
|
|||
|
can expect lower-case encoding names; dropped slicing and offsets
|
|||
|
in the codec APIs
|
|||
|
0.8: added encodings package and raw unicode escape encoding; untabified
|
|||
|
the proposal; added notes on Unicode format strings; added
|
|||
|
.breaklines() method
|
|||
|
0.7: added a whole new set of codec APIs; added a different encoder
|
|||
|
lookup scheme; fixed some names
|
|||
|
0.6: changed "s#" to "t#"; changed <defencbuf> to <defencstr> holding
|
|||
|
a real Python string object; changed Buffer Interface to delegate
|
|||
|
requests to <defencstr>'s buffer interface; removed the explicit
|
|||
|
reference to the unicodec.codecs dictionary (the module can implement
|
|||
|
this in way fit for the purpose); removed the settable default
|
|||
|
encoding; move UnicodeError from unicodec to exceptions; "s#"
|
|||
|
not returns the internal data; passed the UCS-2/UTF-16 checking
|
|||
|
from the Unicode constructor to the Codecs
|
|||
|
0.5: moved sys.bom to unicodec.BOM; added sections on case mapping,
|
|||
|
private use encodings and Unicode character properties
|
|||
|
0.4: added Codec interface, notes on %-formatting, changed some encoding
|
|||
|
details, added comments on stream wrappers, fixed some discussion
|
|||
|
points (most important: Internal Format), clarified the
|
|||
|
'unicode-escape' encoding, added encoding references
|
|||
|
0.3: added references, comments on codec modules, the internal format,
|
|||
|
bf_getcharbuffer and the RE engine; added 'unicode-escape' encoding
|
|||
|
proposed by Tim Peters and fixed repr(u) accordingly
|
|||
|
0.2: integrated Guido's suggestions, added stream codecs and file
|
|||
|
wrapping
|
|||
|
0.1: first version
|
|||
|
|
|||
|
|
|||
|
-----------------------------------------------------------------------------
|
|||
|
Written by Marc-Andre Lemburg, 1999-2000, mal@lemburg.com
|
|||
|
-----------------------------------------------------------------------------
|