cpython/Doc/inst/inst.tex

\documentclass{howto}
\usepackage{ltxmarkup}
\usepackage{times}
\usepackage{distutils}

\title{Installing Python Modules}

% The audience for this document includes people who don't know anything 
% about Python and aren't about to learn the language just in order to
% install and maintain it for their users, i.e. system administrators.
% Thus, I have to be sure to explain the basics at some point:
% sys.path and PYTHONPATH at least.  Should probably give pointers to
% other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc.
% 
% Also, I need to take into account that most modules out there don't
% (yet) use Distutils: briefly explain the old Makefile.pre.in
% convention (maybe move material from the E&E manual to here?), and
% explain where to copy .py and .so files manually if the distribution
% doesn't provide a mechanism for doing so.
%
% Finally, it might be useful to include all the material from my "Care
% and Feeding of a Python Installation" talk in here somewhere.  Yow!

\author{Greg Ward}
\authoraddress{E-mail: \email{gward@python.net}}


\begin{document}

\maketitle

%\begin{abstract}
%\noindent
%Abstract this!
%\end{abstract}

\tableofcontents

\section{Introduction}
\label{sec:intro}

Although Python's extensive standard library covers many programming
needs, there often comes a time when you need to add some new
functionality to your Python installation in the form of third-party
modules.  This might be necessary to support your own programming, or to
support an application that you want to use and that happens to be
written in Python.

In the past, there has been little support for adding third-party
modules to an existing Python installation.  With the introduction of
the Python Distribution Utilities (Distutils for short) in Python 1.6,
this is starting to change.  Not everything will change overnight,
though, so while this document concentrates on installing module
distributions that use the Distutils, we will also spend some time
dealing with the old ways.

This document is aimed primarily at the people who need to install
third-party Python modules: end-users and system administrators who just
need to get some Python application running, and existing Python
programmers who want to add some new goodies to their toolbox.  You
don't need to know Python to read this document; there will be some
brief forays into using Python's interactive mode to explore your
installation, but that's it.  If you're looking for information on how
to distribute your own Python modules so that others may use them, see
the ``Distributing Python Modules'' manual.


\subsection{Best case: trivial installation}
\label{sec:trivial-inst}

In the best case, someone will have prepared a special version of the
module distribution you want to install that is targeted specifically at
your platform and is installed just like any other software on your
platform.  For example, the module developer might make an executable
installer available for Windows users, an RPM package for users of
RPM-based Linux systems (Red Hat, SuSE, Mandrake, and many others), a
Debian package for users of Debian-based Linux systems (Debian proper,
Caldera, Corel, etc.), and so forth.

In that case, you would download the installer appropriate to your
platform and do the obvious thing with it: run it if it's an executable
installer, \code{rpm --install} it if it's an RPM, etc.  You don't need
to run Python or a setup script, you don't need to compile
anything---you might not even need to read any instructions (although
it's always a good idea to do so anyways).

Of course, things will not always be that easy.  You might be interested
in a module distribution that doesn't have an easy-to-use installer for
your platform.  In that case, you'll have to start with the source
distribution released by the module's author/maintainer.  Installing
from a source distribution is not too hard, as long as the modules are
packaged in the standard way.  The bulk of this document is about
building and installing modules from standard source distributions.


\subsection{The new standard: Distutils}
\label{sec:new-standard}

If you download a module source distribution, you can tell pretty
quickly if was packaged and distributed in the standard way, i.e. using
the Distutils.  First, the distribution's name and version number will
be featured prominently in the name of the downloaded archive, e.g.
\file{foo-1.0.tar.gz} or \file{widget-0.9.7.zip}.  Next, the archive
will unpack into a similarly-named directory: \file{foo-1.0} or
\file{widget-0.9.7}.  Additionally, the distribution will contain a
setup script \file{setup.py}, and a \file{README.txt} (or possibly
\file{README}), which should explain that building and installing the
module distribution is a simple matter of running
\begin{verbatim}
python setup.py install
\end{verbatim}

If all these things are true, then you already know how to build and
install the modules you've just downloaded: run the command above.
Unless you need to install things in a non-standard way or customize the
build process, you don't really need this manual.  Or rather, the above
command is everything you need to get out of this manual.


\subsection{The old way: no standards}
\label{sec:old-way}

Before the Distutils, there was no infrastructure to support installing
third-party modules in a consistent, standardized way.  Thus, it's not
really possible to write a general manual for installing Python modules
that don't use the Distutils; the only truly general statement that can
be made is, ``Read the module's own installation instructions.''

However, if such instructions exists at all, they are often woefully
inadequate and targeted at experienced Python developers.  Such users
are already familiar with how the Python library is laid out on their
platform, and know where to copy various files in order for Python to
find them.  This document makes no such assumptions, and explains how
the Python library is laid out on three major platforms (Unix, Windows,
and Mac~OS), so that you can understand what happens when the Distutils
do their job \emph{and} know how to install modules manually when the
module author fails to provide a setup script.

Additionally, while there has not previously been a standard
installation mechanism, Python has had some standard machinery for
building extensions on Unix since Python \XXX{version?}.  This machinery
(the \file{Makefile.pre.in} file) is superseded by the Distutils, but it
will no doubt live on in older module distributions for a while.  This
\file{Makefile.pre.in} mechanism is documented in the ``Extending \&
Embedding Python'' manual, but that manual is aimed at module
developers---hence, we include documentation for builders/installers
here.

All of the pre-Distutils material is tucked away in
section~\ref{sec:pre-distutils}.


\section{Standard Build and Install}
\label{sec:normal-install}

As described in section~\ref{sec:new-standard}, building and installing
a module distribution using the Distutils is usually one simple command:
\begin{verbatim}
python setup.py install
\end{verbatim}
On Unix, you'd run this command from a shell prompt; on Windows, you
have to open a command prompt window and do it there; on Mac~OS ...
\XXX{what the heck do you do on Mac~OS?}.


\subsection{Platform variations}

You should always run the setup command from the distribution root
directory, i.e. the top-level subdirectory that the module source
distribution unpacks into.  For example, if you've just downloaded a
module source distribution \file{foo-1.0.tar.gz} onto a Unix system, the 
normal thing to do is:
\begin{verbatim}
gunzip -c foo-1.0.tar.gz | tar xf -    # unpacks into directory foo-1.0
cd foo-1.0
python setup.py install
\end{verbatim}

On Windows, you'd probably unpack the archive before opening the command
prompt.  If you downloaded the archive file to \file{C:\bslash{}Temp},
then it probably unpacked (depending on your software) into
\file{C:\bslash{}Temp\bslash{}foo-1.0}; from the command prompt window,
you would then run
\begin{verbatim}
cd c:\temp\foo-1.0
python setup.py install
\end{verbatim}

On Mac~OS, ... \XXX{again, how do you run Python scripts on Mac~OS?}

\XXX{arg, my lovely ``bslash'' macro doesn't work in non-tt fonts! help
  me \LaTeX, you're my only hope...}


\subsection{Splitting the job up}

Running \code{setup.py install} builds and installs all modules in one
fell swoop.  If you prefer to work incrementally---especially useful if
you want to customize the build process, or if things are going
wrong---you can use the setup script to do one thing at a time.

For example, you can build everything in one step, and then install
everything in a second step, by invoking the setup script twice:
\begin{verbatim}
python setup.py build
python setup.py install
\end{verbatim}
(If you do this, you will notice that running the \command{install}
command first runs the \command{build} command, which quickly notices
that it has nothing to do, since everything in the \file{build}
directory is up-to-date.)

\XXX{concrete reason for splitting things up?}


\subsection{How building works}

As implied above, the \command{build} command is responsible for putting
the files to install into a \emph{build directory}.  By default, this is
\file{build} under the distribution root; if you're excessively
concerned with speed, or want to keep the source tree pristine, you can
change the build directory with the \option{--build-base} option.  For
example:
\begin{verbatim}
python setup.py build --build-base=/tmp/pybuild/foo-1.0
\end{verbatim}
(Or you could do this permanently with a directive in your system or
personal Distutils configuration file; see
section~\ref{sec:config-files}.)  Normally, this isn't necessary.

The default layout for the build tree is as follows:
\begin{verbatim}
--- build/ --- lib/
or
--- build/ --- lib.<plat>/
               temp.<plat>/
\end{verbatim}
where \code{<plat>} expands to a brief description of the current
OS/hardware platform.  The first form, with just a \file{lib} directory,
is used for ``pure module distributions''---that is, module
distributions that include only pure Python modules.  If a module
distribution contains any extensions (modules written in C/C++, or Java
for JPython), then the second form, with two \code{<plat>} directories,
is used.  In that case, the \file{temp.\filevar{plat}} directory holds
temporary files generated by the compile/link process that don't
actually get installed.  In either case, the \file{lib} (or
\file{lib.\filevar{plat}}) directory contains all Python modules (pure
Python and extensions) that will be installed.

In the future, more directories will be added to handle Python scripts,
documentation, binary executables, and whatever else is needed to handle
the job of installing Python modules and applicatins.


\subsection{How installation works}

After the \command{build} command runs (whether you run it explicitly,
or the \command{install} command does it for you), the work of the
\command{install} command is relatively simple: all it has to do is copy
everything under \file{build/lib} (or \file{build/lib.\filevar{plat}})
to your chosen installation directory.

If you don't choose an installation directory---i.e., if you just run
\code{setup.py install}---then the \command{install} command installs to
the standard location for third-party Python modules.  This location
varies by platform and by how you built/installed Python itself.  On
Unix and Mac OS, it also depends on whether the module distribution
being installed is pure Python or contains extensions (``non-pure''):
\begin{tableiii}{lll}{textrm}%
  {Platform}{Standard installation location}{Default value}
  \lineiii{Unix (pure Python modules)}
          {\file{\filevar{prefix}/lib/python1.6/site-packages}}
          {\file{/usr/local/lib/python1.6/site-packages}}
  \lineiii{Unix (non-pure distribution)}
          {\file{\filevar{exec-prefix}/lib/python1.6/site-packages}}
          {\file{/usr/local/lib/python1.6/site-packages}}
  \lineiii{Windows}
          {\filevar{prefix}}
          {\file{C:\bslash{}Python}
            \footnote{\file{C:\bslash{}Program Files\bslash{}Python}
            under Python 1.6a1 and earlier}}
  \lineiii{Mac~OS (pure Python modules)}
          {\file{\filevar{prefix}}:Lib}
          {\file{Python:Lib}\XXX{???}}
  \lineiii{Mac~OS (non-pure distribution)}
          {\file{\filevar{prefix}}:Mac:PlugIns}
          {\file{Python:Mac:PlugIns}\XXX{???}}
\end{tableiii}
\filevar{prefix} and \filevar{exec-prefix} stand for the directories
that Python is installed to, and where it finds its libraries at
run-time.  They are always the same under Windows and Mac~OS, and very
often the same under Unix.  You can find out what your Python
installation uses for \filevar{prefix} and \filevar{exec-prefix} by
running Python in interactive mode and typing a few simple commands.
Under Unix, just type \code{python} at the shell prompt; under Windows,
run ``Python 1.6 (interpreter)'' \XXX{right?}; under Mac~OS, \XXX{???}.
Once the interpreter is started, you type Python code at the \code{>>>}
prompt.  For example, on my Linux system, I type the three Python
statements shown below, and get the output as shown, to find out my
\filevar{prefix} and \filevar{exec-prefix}:
\begin{verbatim}
Python 1.5.2 (#1, Apr 18 1999, 16:03:16)  [GCC pgcc-2.91.60 19981201 (egcs-1.1.1  on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import sys
>>> sys.prefix
'/usr'
>>> sys.exec_prefix
'/usr'
\end{verbatim}

If you don't want to install to the standard location, or if you don't
have permission to write there, then you need to read about alternate
installations in the next section.


% This rather nasty macro is used to generate the tables that describe
% each installation scheme.  It's nasty because it takes two arguments
% for each "slot" in an installation scheme, there will soon be more
% than five of these slots, and TeX has a limit of 10 arguments to a
% macro.  Uh-oh.

\newcommand{\installscheme}[8]
  {\begin{tableiii}{lll}{textrm}
          {Type of file}
          {Installation Directory}
          {Override option}
     \lineiii{pure module distribution}
             {\filevar{#1}\filenq{#2}}
             {\option{--install-purelib}}
     \lineiii{non-pure module distribution}
             {\filevar{#3}\filenq{#4}}
             {\option{--install-platlib}}
     \lineiii{scripts}
             {\filevar{#5}\filenq{#6}}
             {\option{--install-scripts}}
     \lineiii{data}
             {\filevar{#7}\filenq{#8}}
             {\option{--install-data}}
   \end{tableiii}}

\section{Alternate Installation}
\label{sec:alt-install}

Often, it is necessary or desirable to install modules to a location
other than the standard location for third-party Python modules.  For
example, on a Unix system you might not have permission to write to the
standard third-party module directory.  Or you might wish to try out a
module before making it a standard part of your local Python
installation; this is especially true when upgrading a distribution
already present: you want to make sure your existing base of scripts
still works with the new version before actually upgrading.

The Distutils \command{install} command is designed to make installing
module distributions to an alternate location simple and painless.  The
basic idea is that you supply a base directory for the installation, and
the \command{install} command picks a set of directories (called an
\emph{installation scheme}) under this base directory in which to
install files.  The details differ across platforms, so read whichever
of the following section applies to you.


\subsection{Alternate installation: Unix (the home scheme)}
\label{sec:alt-unix-prefix}

Under Unix, there are two ways to perform an alternate installation.
The ``prefix scheme'' is similar to how alternate installation works
under Windows and Mac~OS, but is not necessarily the most useful way to
maintain a personal Python library.  Hence, we document the more
convenient and commonly useful ``home scheme'' first.

The idea behind the ``home scheme'' is that you build and maintain a
personal stash of Python modules, probably under your home directory.
Installing a new module distribution is as simple as
\begin{verbatim}
python setup.py install --home=<dir>
\end{verbatim}
where you can supply any directory you like for the \option{home}
option.  Lazy typists can just type a tilde (\code{\tilde}); the
\command{install} command will expand this to your home directory:
\begin{verbatim}
python setup.py install --home=~
\end{verbatim}

The \option{home} option defines the installation base directory.  Files
are installed to the following directories under the installation base
as follows:
\installscheme{home}{/lib/python}
              {home}{/lib/python}
              {home}{/bin}
              {home}{/share}

\subsection{Alternate installation: Unix (the prefix scheme)}
\label{sec:alt-unix-home}

The ``prefix scheme'' is useful when you wish to use one Python
installation to perform the build/install (i.e., to run the setup
script), but install modules into the third-party module directory of a
different Python installation (or something that looks like a different
Python installation).  If this sounds a trifle unusual, it is---that's
why the ``home scheme'' comes first.  However, there are at least two
known cases where the prefix scheme will be useful.

First, consider that many Linux distribution put Python in \file{/usr},
rather than the more traditional \file{/usr/local}.  This is entirely
appropriate, since in those cases Python is part of ``the system''
rather than a local add-on.  However, if you are installing Python
modules from source, you probably want them to go in
\file{/usr/local/lib/python1.\filevar{X}} rather than
\file{/usr/lib/python1.\filevar{X}}.  This can be done with
\begin{verbatim}
/usr/bin/python setup.py install --prefix=/usr/local
\end{verbatim}

Another possibility is a network filesystem where the name used to write
to a remote directory is different from the name used to read it: for
example, the Python interpreter accessed as \file{/usr/local/bin/python}
might search for modules in \file{/usr/local/lib/python1.\filevar{X}},
but those modules would have to be installed to, say,
\file{/mnt/\filevar{@server}/export/lib/python1.\filevar{X}}.  This
could be done with
\begin{verbatim}
/usr/local/bin/python setup.py install --prefix=/mnt/@server/export
\end{verbatim}

In either case, the \option{prefix} option defines the installation
base, and the \option{exec-prefix} option defines the platform-specific
installation base, which is used for platform-specific files.
(Currently, this just means non-pure module distributions, but could be
expanded to C libraries, binary executables, etc.)  If
\option{exec-prefix} is not supplied, it defaults to \option{prefix}.
Files are installed as follows:

\installscheme{prefix}{/lib/python1.\filevar{X}/site-packages}
              {exec-prefix}{/lib/python1.\filevar{X}/site-packages}
              {prefix}{/bin}
              {prefix}{/share}

There is no requirement that \option{prefix} or \option{exec-prefix}
actually point to an alternate Python installation; if the directories
listed above do not already exist, they are created at installation
time.

Incidentally, the real reason the prefix scheme is important is simply
that a standard Unix installation uses the prefix scheme, but with
\option{prefix} and \option{exec-prefix} supplied by Python itself (as
\code{sys.prefix} and \code{sys.exec\_prefix}).  Thus, you might think
you'll never use the prefix scheme, but every time you run \code{python
  setup.py install} without any other options, you're using it.

Note that installing extensions to an alternate Python installation has
no effect on how those extensions are built: in particular, the Python
header files (\file{Python.h} and friends) installed with the Python
interpreter used to run the setup script will be used in compiling
extensions.  It is your responsibility to ensure that the interpreter
used to run extensions installed in this way is compatibile with the
interpreter used to build them.  The best way to do this is to ensure
that the two interpreters are the same version of Python (possibly
different builds, or possibly copies of the same build).  (Of course, if
your \option{prefix} and \option{exec-prefix} don't even point to an
alternate Python installation, this is immaterial.)


\subsection{Alternate installation: Windows}
\label{sec:alt-windows}

Since Windows has no conception of a user's home directory, and since
the standard Python installation under Windows is simpler than that
under Unix, there's no point in having separate \option{prefix} and
\option{home} options.  Just use the \option{prefix} option to specify
a base directory, e.g.
\begin{verbatim}
python setup.py install --prefix="\Temp\Python"
\end{verbatim}
to install modules to the \file{\bslash{}Temp} directory on the current
drive.

The installation base is defined by the \option{prefix} option; the
\option{exec-prefix} option is not supported under Windows.  Files are
installed as follows:
\installscheme{prefix}{}
              {prefix}{}
              {prefix}{\bslash{}Scripts}
              {prefix}{\bslash{}Data}


\subsection{Alternate installation: Mac~OS}
\label{sec:alt-macos}

Like Windows, Mac~OS has no notion of home directories (or even of
users), and a fairly simple standard Python installation.  Thus, only a
\option{prefix} option is needed.  It defines the installation base, and 
files are installed under it as follows:

\XXX{how do MacPython users run the interpreter with command-line args?}

\installscheme{prefix}{:Lib}
              {prefix}{:Mac:PlugIns}
              {prefix}{:Scripts}
              {prefix}{:Data}

\XXX{Corran Webster says: ``Modules are found in either \file{:Lib} or
\file{:Mac:Lib}, while extensions usually go in
\file{:Mac:PlugIns}''---does this mean that non-pure distributions should
be divided between \file{:Mac:PlugIns} and \file{:Mac:Lib}?  If so, that
changes the granularity at which we care about modules: instead of
``modules from pure distributions'' and ``modules from non-pure
distributions'', it becomes ``modules from pure distributions'',
``Python modules from non-pure distributions'', and ``extensions from
non-pure distributions''.  Is this necessary?!?}


\section{Custom Installation}
\label{sec:custom-install}

Sometimes, the alternate installation schemes described in
section~\ref{sec:alt-install} just don't do what you want.  You might
want to tweak just one or two directories while keeping everything under
the same base directory, or you might want to completely redefine the
installation scheme.  In either case, you're creating a \emph{custom
  installation scheme}.

You probably noticed the column of ``override options'' in the tables
describing the alternate installation schemes above.  Those options are
how you define a custom installation scheme.  These override options can
be relative, absolute, or explicitly defined in terms of one of the
installation base directories.  (There are two installation base
directories, and they are normally the same---they only differ when you
use the Unix ``prefix scheme'' and supply different \option{prefix} and
\option{exec-prefix} options.)

For example, say you're installing a module distribution to your home
directory under Unix---but you want scripts to go in
\file{\tilde/scripts} rather than \file{\tilde/bin}.  As you might
expect, you can override this directory with the
\option{install-scripts} option; in this case, it makes most sense to
supply a relative path, which will be interpreted relative to the
installation base directory (your home directory, in this case):
\begin{verbatim}
python setup.py install --home --install-scripts=scripts
\end{verbatim}

Another Unix example: suppose your Python installation was built and
installed with a prefix of \file{/usr/local/python}, so under a standard 
installation scripts will wind up in \file{/usr/local/python/bin}.  If
you want them in \file{/usr/local/bin} instead, you would supply this
absolute directory for the \option{install-scripts} option:
\begin{verbatim}
python setup.py install --install-scripts=/usr/local/bin
\end{verbatim}
(This performs an installation using the ``prefix scheme,'' where the
prefix is whatever your Python interpreter was installed with---
\file{/usr/local/python} in this case.)

If you maintain Python on Windows, you might want third-party modules to
live in a subdirectory of \filevar{prefix}, rather than right in
\filevar{prefix} itself.  This is almost as easy as customizing the
script installation directory---you just have to remember that there are
two types of modules to worry about, pure modules and non-pure modules
(i.e., modules from a non-pure distribution).  For example:
\begin{verbatim}
python setup.py install --install-purelib=Site --install-platlib=Site
\end{verbatim}
The specified installation directories are relative to \filevar{prefix}.
Of course, you also have to ensure that these directories are in
Python's module search path, e.g. by putting a \file{.pth} file in
\filevar{prefix} (\XXX{should have a section describing .pth files and
  cross-ref it here}).

If you want to define an entire installation scheme, you just have to
supply all of the installation directory options.  The recommended way
to do this is to supply relative paths; for example, if you want to
maintain all Python module-related files under \file{python} in your
home directory, and you want a separate directory for each platform that
you use your home directory from, you might define the following
installation scheme:
\begin{verbatim}
python setup.py install --home=~ \
                        --install-purelib=python/lib \
                        --install-platlib=python/lib.$PLAT \
                        --install-scripts=python/scripts
                        --install-data=python/data
\end{verbatim}
or, equivalently,
\begin{verbatim}
python setup.py install --home=~/python \
                        --install-purelib=lib \
                        --install-platlib=lib.$PLAT \
                        --install-scripts=scripts
                        --install-data=data
\end{verbatim}
\code{\$PLAT} is not (necessarily) an environment variable---it will be
expanded by the Distutils as it parses your command line options (just
as it does when parsing your configuration file(s)).

Obviously, specifying the entire installation scheme every time you
install a new module distribution would be very tedious.  Thus, you can
put these options into your Distutils config file (see
section~\ref{sec:config-files}):
\begin{verbatim}
[install]
install-base=$HOME
install-purelib=python/lib
install-platlib=python/lib.$PLAT
install-scripts=python/scripts
install-data=python/data
\end{verbatim}
or, equivalently,
\begin{verbatim}
[install]
install-base=$HOME/python
install-purelib=lib
install-platlib=lib.$PLAT
install-scripts=scripts
install-data=data
\end{verbatim}
Note that these two are \emph{not} equivalent if you supply a different
installation base directory when you run the setup script.  For example,
\begin{verbatim}
python setup.py --install-base=/tmp
\end{verbatim}
would install pure modules to \filevar{/tmp/python/lib} in the first
case, and to \filevar{/tmp/lib} in the second case.  (For the second
case, you probably want to supply an installation base of
\file{/tmp/python}.)

You probably noticed the use of \code{\$HOME} and \code{\$PLAT} in the
sample configuration file input.  These are Distutils configuration
variables, which bear a strong resemblance to environment variables.  In
fact, you can use environment variables in config files---on platforms
that have such a notion---but the Distutils additionally define a few
extra variables that may not be in your environment, such as
\code{\$PLAT}.  (And of course, you can only use the configuration
variables supplied by the Distutils on systems that don't have
environment variables, such as Mac~OS (\XXX{true?}).)  See
section~\ref{sec:config-files} for details.

\XXX{need some Windows and Mac~OS examples---when would custom
  installation schemes be needed on those platforms?}


\section{Distutils Configuration Files}
\label{sec:config-files}

\XXX{not even implemented yet, much less documented!}


\section{Pre-Distutils Conventions}
\label{sec:pre-distutils}


\subsection{The Makefile.pre.in file}
\label{sec:makefile-pre-in}


\subsection{Installing modules manually}
\label{sec:manual-install}



\end{document}