1994-01-02 01:22:07 +00:00
|
|
|
\section{Built-in Module \sectcode{regex}}
|
|
|
|
|
|
|
|
\bimodindex{regex}
|
|
|
|
This module provides regular expression matching operations similar to
|
|
|
|
those found in Emacs. It is always available.
|
|
|
|
|
1996-10-24 22:49:13 +00:00
|
|
|
By default the patterns are Emacs-style regular expressions
|
|
|
|
(with one exception). There is
|
1994-01-02 01:22:07 +00:00
|
|
|
a way to change the syntax to match that of several well-known
|
1995-08-11 00:31:57 +00:00
|
|
|
\UNIX{} utilities. The exception is that Emacs' \samp{\e s}
|
|
|
|
pattern is not supported, since the original implementation references
|
|
|
|
the Emacs syntax tables.
|
1994-01-02 01:22:07 +00:00
|
|
|
|
|
|
|
This module is 8-bit clean: both patterns and strings may contain null
|
|
|
|
bytes and characters whose high bit is set.
|
|
|
|
|
1994-01-03 00:00:31 +00:00
|
|
|
\strong{Please note:} There is a little-known fact about Python string
|
|
|
|
literals which means that you don't usually have to worry about
|
|
|
|
doubling backslashes, even though they are used to escape special
|
|
|
|
characters in string literals as well as in regular expressions. This
|
|
|
|
is because Python doesn't remove backslashes from string literals if
|
|
|
|
they are followed by an unrecognized escape character.
|
|
|
|
\emph{However}, if you want to include a literal \dfn{backslash} in a
|
|
|
|
regular expression represented as a string literal, you have to
|
1995-03-07 10:14:09 +00:00
|
|
|
\emph{quadruple} it. E.g.\ to extract \LaTeX\ \samp{\e section\{{\rm
|
1994-01-03 00:00:31 +00:00
|
|
|
\ldots}\}} headers from a document, you can use this pattern:
|
1996-06-26 19:43:22 +00:00
|
|
|
\code{'\e \e \e \e section\{\e (.*\e )\}'}. \emph{Another exception:}
|
|
|
|
the escape sequece \samp{\e b} is significant in string literals
|
|
|
|
(where it means the ASCII bell character) as well as in Emacs regular
|
|
|
|
expressions (where it stands for a word boundary), so in order to
|
|
|
|
search for a word boundary, you should use the pattern \code{'\e \e b'}.
|
|
|
|
Similarly, a backslash followed by a digit 0-7 should be doubled to
|
|
|
|
avoid interpretation as an octal escape.
|
|
|
|
|
|
|
|
\subsection{Regular Expressions}
|
|
|
|
|
|
|
|
A regular expression (or RE) specifies a set of strings that matches
|
|
|
|
it; the functions in this module let you check if a particular string
|
1996-10-24 22:49:13 +00:00
|
|
|
matches a given regular expression (or if a given regular expression
|
|
|
|
matches a particular string, which comes down to the same thing).
|
1996-06-26 19:43:22 +00:00
|
|
|
|
|
|
|
Regular expressions can be concatenated to form new regular
|
|
|
|
expressions; if \emph{A} and \emph{B} are both regular expressions,
|
|
|
|
then \emph{AB} is also an regular expression. If a string \emph{p}
|
|
|
|
matches A and another string \emph{q} matches B, the string \emph{pq}
|
|
|
|
will match AB. Thus, complex expressions can easily be constructed
|
|
|
|
from simpler ones like the primitives described here. For details of
|
|
|
|
the theory and implementation of regular expressions, consult almost
|
|
|
|
any textbook about compiler construction.
|
|
|
|
|
|
|
|
% XXX The reference could be made more specific, say to
|
|
|
|
% "Compilers: Principles, Techniques and Tools", by Alfred V. Aho,
|
|
|
|
% Ravi Sethi, and Jeffrey D. Ullman, or some FA text.
|
|
|
|
|
1996-10-24 22:49:13 +00:00
|
|
|
A brief explanation of the format of regular expressions follows.
|
1996-06-26 19:43:22 +00:00
|
|
|
|
|
|
|
Regular expressions can contain both special and ordinary characters.
|
|
|
|
Ordinary characters, like '\code{A}', '\code{a}', or '\code{0}', are
|
|
|
|
the simplest regular expressions; they simply match themselves. You
|
|
|
|
can concatenate ordinary characters, so '\code{last}' matches the
|
1996-10-24 22:49:13 +00:00
|
|
|
characters 'last'. (In the rest of this section, we'll write RE's in
|
|
|
|
\code{this special font}, usually without quotes, and strings to be
|
|
|
|
matched 'in single quotes'.)
|
1996-06-26 19:43:22 +00:00
|
|
|
|
|
|
|
Special characters either stand for classes of ordinary characters, or
|
|
|
|
affect how the regular expressions around them are interpreted.
|
|
|
|
|
|
|
|
The special characters are:
|
|
|
|
\begin{itemize}
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{.}] (Dot.) Matches any character except a newline.
|
|
|
|
\item[\code{\^}] (Caret.) Matches the start of the string.
|
|
|
|
\item[\code{\$}] Matches the end of the string.
|
1996-06-26 19:43:22 +00:00
|
|
|
\code{foo} matches both 'foo' and 'foobar', while the regular
|
1996-12-13 22:04:31 +00:00
|
|
|
expression '\code{foo\$}' matches only 'foo'.
|
1996-06-26 19:43:22 +00:00
|
|
|
\item[\code{*}] Causes the resulting RE to
|
|
|
|
match 0 or more repetitions of the preceding RE. \code{ab*} will
|
|
|
|
match 'a', 'ab', or 'a' followed by any number of 'b's.
|
|
|
|
\item[\code{+}] Causes the
|
|
|
|
resulting RE to match 1 or more repetitions of the preceding RE.
|
|
|
|
\code{ab+} will match 'a' followed by any non-zero number of 'b's; it
|
|
|
|
will not match just 'a'.
|
|
|
|
\item[\code{?}] Causes the resulting RE to
|
|
|
|
match 0 or 1 repetitions of the preceding RE. \code{ab?} will
|
|
|
|
match either 'a' or 'ab'.
|
|
|
|
|
|
|
|
\item[\code{\e}] Either escapes special characters (permitting you to match
|
|
|
|
characters like '*?+\&\$'), or signals a special sequence; special
|
|
|
|
sequences are discussed below. Remember that Python also uses the
|
|
|
|
backslash as an escape sequence in string literals; if the escape
|
|
|
|
sequence isn't recognized by Python's parser, the backslash and
|
|
|
|
subsequent character are included in the resulting string. However,
|
|
|
|
if Python would recognize the resulting sequence, the backslash should
|
|
|
|
be repeated twice.
|
|
|
|
|
|
|
|
\item[\code{[]}] Used to indicate a set of characters. Characters can
|
|
|
|
be listed individually, or a range is indicated by giving two
|
|
|
|
characters and separating them by a '-'. Special characters are
|
|
|
|
not active inside sets. For example, \code{[akm\$]}
|
|
|
|
will match any of the characters 'a', 'k', 'm', or '\$'; \code{[a-z]} will
|
|
|
|
match any lowercase letter.
|
|
|
|
|
|
|
|
If you want to include a \code{]} inside a
|
|
|
|
set, it must be the first character of the set; to include a \code{-},
|
|
|
|
place it as the first or last character.
|
|
|
|
|
|
|
|
Characters \emph{not} within a range can be matched by including a
|
|
|
|
\code{\^} as the first character of the set; \code{\^} elsewhere will
|
|
|
|
simply match the '\code{\^}' character.
|
|
|
|
\end{itemize}
|
|
|
|
|
|
|
|
The special sequences consist of '\code{\e}' and a character
|
|
|
|
from the list below. If the ordinary character is not on the list,
|
|
|
|
then the resulting RE will match the second character. For example,
|
|
|
|
\code{\e\$} matches the character '\$'. Ones where the backslash
|
|
|
|
should be doubled are indicated.
|
|
|
|
|
|
|
|
\begin{itemize}
|
|
|
|
\item[\code{\e|}]\code{A\e|B}, where A and B can be arbitrary REs,
|
1996-10-24 22:49:13 +00:00
|
|
|
creates a regular expression that will match either A or B. This can
|
|
|
|
be used inside groups (see below) as well.
|
1996-06-26 19:43:22 +00:00
|
|
|
%
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e( \e)}] Indicates the start and end of a group; the
|
1996-06-26 19:43:22 +00:00
|
|
|
contents of a group can be matched later in the string with the
|
1996-12-13 22:04:31 +00:00
|
|
|
\code{\e [1-9]} special sequence, described next.
|
1996-06-26 19:43:22 +00:00
|
|
|
%
|
|
|
|
{\fulllineitems\item[\code{\e \e 1, ... \e \e 7, \e 8, \e 9}]
|
1996-12-13 22:04:31 +00:00
|
|
|
Matches the contents of the group of the same
|
1996-06-26 19:43:22 +00:00
|
|
|
number. For example, \code{\e (.+\e ) \e \e 1} matches 'the the' or
|
|
|
|
'55 55', but not 'the end' (note the space after the group). This
|
|
|
|
special sequence can only be used to match one of the first 9 groups;
|
|
|
|
groups with higher numbers can be matched using the \code{\e v}
|
1996-10-24 22:49:13 +00:00
|
|
|
sequence. (\code{\e 8} and \code{\e 9} don't need a double backslash
|
1996-12-13 22:04:31 +00:00
|
|
|
because they are not octal digits.)}
|
1996-06-26 19:43:22 +00:00
|
|
|
%
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e \e b}] Matches the empty string, but only at the
|
1996-06-26 19:43:22 +00:00
|
|
|
beginning or end of a word. A word is defined as a sequence of
|
|
|
|
alphanumeric characters, so the end of a word is indicated by
|
1996-12-13 22:04:31 +00:00
|
|
|
whitespace or a non-alphanumeric character.
|
1996-06-26 19:43:22 +00:00
|
|
|
%
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e B}] Matches the empty string, but when it is \emph{not} at the
|
|
|
|
beginning or end of a word.
|
1996-06-26 19:43:22 +00:00
|
|
|
%
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e v}] Must be followed by a two digit decimal number, and
|
|
|
|
matches the contents of the group of the same number. The group number must be between 1 and 99, inclusive.
|
1996-06-26 19:43:22 +00:00
|
|
|
%
|
|
|
|
\item[\code{\e w}]Matches any alphanumeric character; this is
|
|
|
|
equivalent to the set \code{[a-zA-Z0-9]}.
|
|
|
|
%
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e W}] Matches any non-alphanumeric character; this is
|
|
|
|
equivalent to the set \code{[\^a-zA-Z0-9]}.
|
|
|
|
\item[\code{\e <}] Matches the empty string, but only at the beginning of a
|
1996-06-26 19:43:22 +00:00
|
|
|
word. A word is defined as a sequence of alphanumeric characters, so
|
|
|
|
the end of a word is indicated by whitespace or a non-alphanumeric
|
1996-12-13 22:04:31 +00:00
|
|
|
character.
|
|
|
|
\item[\code{\e >}] Matches the empty string, but only at the end of a
|
|
|
|
word.
|
1996-06-26 19:43:22 +00:00
|
|
|
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e \e \e \e}] Matches a literal backslash.
|
1996-10-24 22:49:13 +00:00
|
|
|
|
1996-06-26 19:43:22 +00:00
|
|
|
% In Emacs, the following two are start of buffer/end of buffer. In
|
|
|
|
% Python they seem to be synonyms for ^$.
|
1996-12-13 22:04:31 +00:00
|
|
|
\item[\code{\e `}] Like \code{\^}, this only matches at the start of the
|
|
|
|
string.
|
1996-06-26 19:43:22 +00:00
|
|
|
\item[\code{\e \e '}] Like \code{\$}, this only matches at the end of the
|
|
|
|
string.
|
|
|
|
% end of buffer
|
|
|
|
\end{itemize}
|
|
|
|
|
|
|
|
\subsection{Module Contents}
|
1994-01-02 01:22:07 +00:00
|
|
|
|
|
|
|
The module defines these functions, and an exception:
|
|
|
|
|
|
|
|
\renewcommand{\indexsubitem}{(in module regex)}
|
1994-01-03 00:00:31 +00:00
|
|
|
|
1994-01-02 01:22:07 +00:00
|
|
|
\begin{funcdesc}{match}{pattern\, string}
|
|
|
|
Return how many characters at the beginning of \var{string} match
|
|
|
|
the regular expression \var{pattern}. Return \code{-1} if the
|
|
|
|
string does not match the pattern (this is different from a
|
|
|
|
zero-length match!).
|
|
|
|
\end{funcdesc}
|
|
|
|
|
|
|
|
\begin{funcdesc}{search}{pattern\, string}
|
|
|
|
Return the first position in \var{string} that matches the regular
|
1996-10-24 22:49:13 +00:00
|
|
|
expression \var{pattern}. Return \code{-1} if no position in the string
|
1994-01-02 01:22:07 +00:00
|
|
|
matches the pattern (this is different from a zero-length match
|
|
|
|
anywhere!).
|
|
|
|
\end{funcdesc}
|
|
|
|
|
1994-08-08 12:30:22 +00:00
|
|
|
\begin{funcdesc}{compile}{pattern\optional{\, translate}}
|
1994-01-02 01:22:07 +00:00
|
|
|
Compile a regular expression pattern into a regular expression
|
|
|
|
object, which can be used for matching using its \code{match} and
|
1995-03-17 16:07:09 +00:00
|
|
|
\code{search} methods, described below. The optional argument
|
1994-01-02 01:22:07 +00:00
|
|
|
\var{translate}, if present, must be a 256-character string
|
|
|
|
indicating how characters (both of the pattern and of the strings to
|
|
|
|
be matched) are translated before comparing them; the \code{i}-th
|
|
|
|
element of the string gives the translation for the character with
|
1995-03-17 16:07:09 +00:00
|
|
|
\ASCII{} code \code{i}. This can be used to implement
|
|
|
|
case-insensitive matching; see the \code{casefold} data item below.
|
1994-01-02 01:22:07 +00:00
|
|
|
|
|
|
|
The sequence
|
|
|
|
|
|
|
|
\bcode\begin{verbatim}
|
|
|
|
prog = regex.compile(pat)
|
|
|
|
result = prog.match(str)
|
|
|
|
\end{verbatim}\ecode
|
|
|
|
|
|
|
|
is equivalent to
|
|
|
|
|
|
|
|
\bcode\begin{verbatim}
|
|
|
|
result = regex.match(pat, str)
|
|
|
|
\end{verbatim}\ecode
|
|
|
|
|
|
|
|
but the version using \code{compile()} is more efficient when multiple
|
|
|
|
regular expressions are used concurrently in a single program. (The
|
|
|
|
compiled version of the last pattern passed to \code{regex.match()} or
|
|
|
|
\code{regex.search()} is cached, so programs that use only a single
|
|
|
|
regular expression at a time needn't worry about compiling regular
|
|
|
|
expressions.)
|
|
|
|
\end{funcdesc}
|
|
|
|
|
|
|
|
\begin{funcdesc}{set_syntax}{flags}
|
|
|
|
Set the syntax to be used by future calls to \code{compile},
|
|
|
|
\code{match} and \code{search}. (Already compiled expression objects
|
|
|
|
are not affected.) The argument is an integer which is the OR of
|
|
|
|
several flag bits. The return value is the previous value of
|
|
|
|
the syntax flags. Names for the flags are defined in the standard
|
|
|
|
module \code{regex_syntax}; read the file \file{regex_syntax.py} for
|
|
|
|
more information.
|
|
|
|
\end{funcdesc}
|
|
|
|
|
1994-08-08 12:30:22 +00:00
|
|
|
\begin{funcdesc}{symcomp}{pattern\optional{\, translate}}
|
1994-01-03 00:00:31 +00:00
|
|
|
This is like \code{compile}, but supports symbolic group names: if a
|
1995-03-07 10:14:09 +00:00
|
|
|
parenthesis-enclosed group begins with a group name in angular
|
1994-01-03 00:00:31 +00:00
|
|
|
brackets, e.g. \code{'\e(<id>[a-z][a-z0-9]*\e)'}, the group can
|
|
|
|
be referenced by its name in arguments to the \code{group} method of
|
|
|
|
the resulting compiled regular expression object, like this:
|
1995-02-27 17:52:35 +00:00
|
|
|
\code{p.group('id')}. Group names may contain alphanumeric characters
|
|
|
|
and \code{'_'} only.
|
1994-01-03 00:00:31 +00:00
|
|
|
\end{funcdesc}
|
|
|
|
|
1994-01-02 01:22:07 +00:00
|
|
|
\begin{excdesc}{error}
|
|
|
|
Exception raised when a string passed to one of the functions here
|
|
|
|
is not a valid regular expression (e.g., unmatched parentheses) or
|
|
|
|
when some other error occurs during compilation or matching. (It is
|
|
|
|
never an error if a string contains no match for a pattern.)
|
|
|
|
\end{excdesc}
|
|
|
|
|
|
|
|
\begin{datadesc}{casefold}
|
|
|
|
A string suitable to pass as \var{translate} argument to
|
|
|
|
\code{compile} to map all upper case characters to their lowercase
|
|
|
|
equivalents.
|
|
|
|
\end{datadesc}
|
|
|
|
|
|
|
|
\noindent
|
|
|
|
Compiled regular expression objects support these methods:
|
|
|
|
|
|
|
|
\renewcommand{\indexsubitem}{(regex method)}
|
1994-08-08 12:30:22 +00:00
|
|
|
\begin{funcdesc}{match}{string\optional{\, pos}}
|
1994-01-02 01:22:07 +00:00
|
|
|
Return how many characters at the beginning of \var{string} match
|
|
|
|
the compiled regular expression. Return \code{-1} if the string
|
|
|
|
does not match the pattern (this is different from a zero-length
|
|
|
|
match!).
|
|
|
|
|
|
|
|
The optional second parameter \var{pos} gives an index in the string
|
|
|
|
where the search is to start; it defaults to \code{0}. This is not
|
|
|
|
completely equivalent to slicing the string; the \code{'\^'} pattern
|
|
|
|
character matches at the real begin of the string and at positions
|
|
|
|
just after a newline, not necessarily at the index where the search
|
|
|
|
is to start.
|
|
|
|
\end{funcdesc}
|
|
|
|
|
1994-08-08 12:30:22 +00:00
|
|
|
\begin{funcdesc}{search}{string\optional{\, pos}}
|
1994-01-02 01:22:07 +00:00
|
|
|
Return the first position in \var{string} that matches the regular
|
|
|
|
expression \code{pattern}. Return \code{-1} if no position in the
|
|
|
|
string matches the pattern (this is different from a zero-length
|
|
|
|
match anywhere!).
|
|
|
|
|
|
|
|
The optional second parameter has the same meaning as for the
|
|
|
|
\code{match} method.
|
|
|
|
\end{funcdesc}
|
|
|
|
|
|
|
|
\begin{funcdesc}{group}{index\, index\, ...}
|
|
|
|
This method is only valid when the last call to the \code{match}
|
|
|
|
or \code{search} method found a match. It returns one or more
|
|
|
|
groups of the match. If there is a single \var{index} argument,
|
|
|
|
the result is a single string; if there are multiple arguments, the
|
|
|
|
result is a tuple with one item per argument. If the \var{index} is
|
|
|
|
zero, the corresponding return value is the entire matching string; if
|
1994-01-03 00:00:31 +00:00
|
|
|
it is in the inclusive range [1..99], it is the string matching the
|
1994-01-02 01:22:07 +00:00
|
|
|
the corresponding parenthesized group (using the default syntax,
|
|
|
|
groups are parenthesized using \code{\\(} and \code{\\)}). If no
|
|
|
|
such group exists, the corresponding result is \code{None}.
|
1994-01-03 00:00:31 +00:00
|
|
|
|
|
|
|
If the regular expression was compiled by \code{symcomp} instead of
|
|
|
|
\code{compile}, the \var{index} arguments may also be strings
|
|
|
|
identifying groups by their group name.
|
1994-01-02 01:22:07 +00:00
|
|
|
\end{funcdesc}
|
|
|
|
|
|
|
|
\noindent
|
|
|
|
Compiled regular expressions support these data attributes:
|
|
|
|
|
|
|
|
\renewcommand{\indexsubitem}{(regex attribute)}
|
1994-01-03 00:00:31 +00:00
|
|
|
|
1994-01-02 01:22:07 +00:00
|
|
|
\begin{datadesc}{regs}
|
|
|
|
When the last call to the \code{match} or \code{search} method found a
|
|
|
|
match, this is a tuple of pairs of indices corresponding to the
|
|
|
|
beginning and end of all parenthesized groups in the pattern. Indices
|
|
|
|
are relative to the string argument passed to \code{match} or
|
|
|
|
\code{search}. The 0-th tuple gives the beginning and end or the
|
|
|
|
whole pattern. When the last match or search failed, this is
|
|
|
|
\code{None}.
|
|
|
|
\end{datadesc}
|
|
|
|
|
|
|
|
\begin{datadesc}{last}
|
|
|
|
When the last call to the \code{match} or \code{search} method found a
|
|
|
|
match, this is the string argument passed to that method. When the
|
|
|
|
last match or search failed, this is \code{None}.
|
|
|
|
\end{datadesc}
|
|
|
|
|
|
|
|
\begin{datadesc}{translate}
|
|
|
|
This is the value of the \var{translate} argument to
|
|
|
|
\code{regex.compile} that created this regular expression object. If
|
|
|
|
the \var{translate} argument was omitted in the \code{regex.compile}
|
|
|
|
call, this is \code{None}.
|
|
|
|
\end{datadesc}
|
1994-01-03 00:00:31 +00:00
|
|
|
|
|
|
|
\begin{datadesc}{givenpat}
|
|
|
|
The regular expression pattern as passed to \code{compile} or
|
|
|
|
\code{symcomp}.
|
|
|
|
\end{datadesc}
|
|
|
|
|
|
|
|
\begin{datadesc}{realpat}
|
|
|
|
The regular expression after stripping the group names for regular
|
|
|
|
expressions compiled with \code{symcomp}. Same as \code{givenpat}
|
|
|
|
otherwise.
|
|
|
|
\end{datadesc}
|
|
|
|
|
|
|
|
\begin{datadesc}{groupindex}
|
|
|
|
A dictionary giving the mapping from symbolic group names to numerical
|
|
|
|
group indices for regular expressions compiled with \code{symcomp}.
|
|
|
|
\code{None} otherwise.
|
|
|
|
\end{datadesc}
|