Installation¶
This document describes the necessary steps to configure Patchwork in a development environment. If you are interested in deploying Patchwork in a production environment, refer to the deployment guide instead.
To begin, you should clone Patchwork:
$ git clone git://github.com/getpatchwork/patchwork.git
Docker-Based Installation¶
Patchwork provides a Docker-based environment for quick configuration of a development environment. This is the preferred installation method. To configure Patchwork using Docker:
Install docker and docker-compose.
Create a
.env
file in the root directory of the project and store yourUID
andGID
attribute there.$ echo "UID=$UID" > .env $ echo "GID=`id -g`" >> .env
Build the images. This will download over 200MB from the internet:
$ docker-compose build
To use Postgres instead of MySQL, give the
-f docker-compose-pg.yml
argument todocker-compose
. This is required on non-x86 architectures as the MySQL Docker images do not have multiarch support.Run
docker-compose up
:$ docker-compose up
This will be visible at http://localhost:8000/.
To run a shell within this environment, run:
$ docker-compose run --rm web --shell
To run django-manage
commands, such as createsuperuser
or migrate
,
run:
$ docker-compose run --rm web python manage.py createsuperuser
To access the SQL command-line client, run:
$ docker-compose run --rm web python manage.py dbshell
To backup the database, run:
$ docker-compose run --rm web python manage.py dbbackup
Likewise, to restore an older version of the database, run:
$ docker-compose run --rm -web python manage.py dbrestore
To run unit tests against the system Python packages, run:
$ docker-compose run --rm web python manage.py test
To run unit tests for multiple versions using tox
, run:
$ docker-compose run --rm web tox
To reset the database before any of these commands, add --reset
to the
command line after web
and before any other arguments:
$ docker-compose run --rm web --reset tox
Any local edits to the project files made locally are immediately visible to the Docker container, and so should be picked up by the Django auto-reloader.
For more information on Docker itself, please refer to the docker and docker-compose documentation.
Note
If using SELinux, you will need to create a custom SELinux rule to allow the Docker process to access your working directory. Run:
$ chcon -RT svirt_sandbox_file_t $PATCHWORK_DIR
where $PATCHWORK_DIR
is the absolute patch to the patchwork
folder
created when you cloned the repo. For more information, see man docker
run
.
Note
If you see an error like the below:
ERROR: Couldn't connect to the Docker daemon at http+docker://localunixsocket - is it running?
ensure you have correctly installed Docker, added your user to the
docker
group, and started the daemon, per the docker documentation.
Note
If you see an error like the below:
You must define UID in .env
Ensure you have created a .env
file in the root of your project
directory and stored the UID
attribute there. For more information on
why this is necessary, refer to this docker-compose issue.
Manual Installation¶
Manual installation can be used where use of Docker is not possible or desired.
Install Required Packages¶
There are a number of different requirements for developing Patchwork:
- Python and libraries
- A supported database (RDBMS)
These are detailed below.
Python Requirements¶
To develop Python-based software you first need Python. Patchwork supports both
Python 2.7 and Python 3.5+. One of these will be installed by default on many
installations, though they can also be installed manually using the python
or python3
packages.
It’s a good idea to use virtual environments to develop Python software. Virtual environments are “instances” of your system Python without any of the additional Python packages installed. They are useful to develop and possibly deploy Patchwork against a “well known” set of dependencies, but they can also be used to test Patchwork against several versions of Django.
If you do not have virtualenv
installed then you should install it now. This
can be installed using the python-virtualenv
or python3-virtualenv
packages. Alternatively you can install these using pip
.
It is also helpful to install tox
which is used for running tests in
Patchwork. This can be installed using the python-tox
or python3-tox
packages, or via pip
.
Database Requirements¶
If not already installed, you may need to install an RDBMS. You can use either
MariaDB/MySQL or PostgreSQL for this purpose. You should also install the
development headers, known as libmysqlclient-dev
or libpq-dev
respectively on Debian-based Debian-based distros like Ubuntu and
mysql-devel
or postgresql-devel
on RHEL-based distros.
Note
While Django provides support for multiple database backends, Patchwork itself is only tested against MySQL/MariaDB and PostgreSQL. Should you wish to use a different backend, ensure you validate this first (and perhaps upstream any changes you may find necessary).
Note
You may be tempted to use SQLite to develop Patchwork. We’d advise against doing this. SQLite supports a subset of the functionality of “full” RDBMS like MySQL: for example, case-sensitive matching of Unicode is not supported. You will find some tests provided by Patchwork fail and some patches you develop may fail in production due to these differences.
Example Installation¶
An example for installing all these packages and the MySQL RDBMS on Ubuntu 15.04 is given below:
$ sudo apt-get install python python-pip python-dev python-virtualenv \
python-tox mysql-server libmysqlclient-dev
If you have an existing MariaDB/MySQL installation and have installed pip
already/are using Python 3.5+ then you can install all packages using pip
:
$ sudo pip install virtualenv tox
If you wish to use Python 3 then simply replace python
with python3
in
the above command.
Configure Virtual Environment¶
Note
If you are interested in simply testing Patchwork, many of the below steps are not required. tox will automatically install dependencies and use virtual environments when testing.
Once these requirements are installed, you should create and activate a new virtual environment. This can be done like so:
$ virtualenv .venv
$ source .venv/bin/activate
(.venv)$
Note
If you installed a Python 3.x-based virtual environment package, adjust the
executable indicated above as necessary, e.g. virtualenv-3.7
.
Now install the packages. Patchwork provides three requirements files.
requirements-dev.txt
- Packages required to configure a development environment
requirements-prod.txt
- Packages required for deploying Patchwork in production
requirements-test.txt
- Packages required to run tests
We’re going to install the first of these, which can be done like so:
(.venv)$ cd patchwork
(.venv)$ pip install -r requirements-dev.txt
Note
Once configured this does not need to be done again unless the requirements change, e.g. Patchwork requires an updated version of Django.
Initialize the Database¶
One installed, the database must be configured. We will assume you have root access to the database for these steps.
To begin, export your database credentials as follows:
(.venv)$ db_user=root
(.venv)$ db_pass=password
Now, create the database. If this is your first time configuring the database,
you must create a patchwork
user (or similar) along with the database
instance itself. The commands below will do this, dropping existing databases
if necessary:
(.venv)$ mysql -u$db_user -p$db_pass << EOF
DROP DATABASE IF EXISTS patchwork;
CREATE DATABASE patchwork CHARACTER SET utf8;
GRANT ALL PRIVILEGES ON patchwork.* TO 'patchwork'@'localhost'
IDENTIFIED BY 'password';
EOF
Note
The patchwork
username and password
password are the defaults
expected by the provided dev
settings files. If using something
different, export the PW_TEST_DB_USER
and PW_TEST_DB_PASS
variables
described in the Environment Variables section below.
Alternatively, you can create your own settings file with these variables
hardcoded and change the value of DJANGO_SETTINGS_MODULE
as described
below.
Load Initial Data¶
Before continuing, we need to tell Django where it can find our configuration.
Patchwork provides a default development settings.py
file for this purpose.
To use this, export the DJANGO_SETTINGS_MODULE
environment variable as
described below:
(.venv)$ export DJANGO_SETTINGS_MODULE=patchwork.settings.dev
Alternatively you can provide your own settings.py
file and provide the path
to that instead.
Once done, we need to create the tables in the database. This can be done using
the migrate
command of the manage.py
executable:
(.venv)$ ./manage.py migrate
Next, you should load the initial fixtures into Patchwork. These initial fixtures provide.
default_tags.xml
- The tags that Patchwork will extract from mails. For example:
Acked-By
,Reviewed-By
default_states.xml
- The states that a patch can be in. For example:
Accepted
,Rejected
default_projects.xml
- A default project that you can then upload patches for
These can be loaded using the loaddata
command:
(.venv)$ ./manage.py loaddata default_tags default_states default_projects
You should also take the opportunity to create a “superuser”. You can do this
using the aptly-named createsuperuser
command:
(.venv)$ ./manage.py createsuperuser
Import Mailing List Archives¶
Regardless of your installation method of choice, you will probably want to
load some real emails into the system. This can be done manually, however it’s
generally much easier to download an archive from a Mailman instance and load
these using the parsearchive
command. You can do this like so:
(.venv)$ mm_user=<myusername>
(.venv)$ mm_pass=<mypassword>
(.venv)$ mm_host=https://lists.ozlabs.org
(.venv)$ mm_url=$mm_host/private/patchwork.mbox/patchwork.mbox
(.venv)$ curl -F username=$mm_user -F password=$mm_pass -k -O $mm_url
where mm_user
and mm_pass
are the username and password you have
registered with on the Mailman instance found at mm_host
.
Note
We provide instructions for downloading archives from the Patchwork mailing
list, but almost any instance of Mailman will allow downloading of archives
as seen above; simply change the pw_url
variable defined. You can find
more informations about this here.
Load these archives into Patchwork. Depending on the size of the downloaded archives this may take some time:
(.venv)$ ./manage.py parsearchive patchwork.mbox
Finally, run the server and browse to the IP address of your board using your browser of choice:
(.venv)$ ./manage.py runserver 0.0.0.0:8000
Once finished, you can kill the server (Ctrl+C) and exit the virtual environment:
(.venv)$ deactivate
$
Should you wish to re-enter this environment, simply source the activate
script again.
Django Debug Toolbar¶
Patchwork installs and enables the ‘Django Debug Toolbar’ application by default when using development settings and requirements. This provides a configurable set of panels that display various debug information about the current request/response and, when clicked, display more details about the panel’s content.
Important
By default, the toolbar is only displayed if you are developing on
localhost
. If developing on a different machine, you should configure
an SSH tunnel such that, for example, localhost:8000
points to
[DEV_MACHINE_IP]:8000
.
For more information, refer to the documentation.
Django Database Backup¶
Patchwork installs and enables the ‘Django Database Backup’ application by default when using development settings and requirements. This provides the following management commands, which can be useful for hacking on Patchwork:
dbbackup
dbrestore
mediabackup
mediarestore
For more information, refer to the documentation.
Environment Variables¶
The following environment variables are available to configure settings when
using the provided dev
settings file.
PW_TEST_DB_NAME=patchwork
- Name of the database
PW_TEST_DB_USER=patchwork
- Username to access the database with
PW_TEST_DB_PASS=password
- Password to access the database with<
PW_TEST_DB_TYPE=mysql
- Type of database to use. Options:
mysql
,postgres