Django 2 with Apache wsgi

From this DigitalOcean tutorial.

To complete our system setup we will require 3 additional packages, as we intend to use Django 2+, we'll install the required packages for python 3, let's install them:

> apt-get update
> apt-get install python3-pip apache2 libapache2-mod-wsgi-py3

We need to identify the apache runtime user id and guid for later mounting our shared folder:

> id www-data
uid=33(www-data) gid=33(www-data) groups=33(www-data)

VirtualBox has chosen not to allow symlinks to work in shared folders, you can learn more on the subject by reading this virtualbox ticket.

This is obviously a major problem as this will prevent you from deploying python virtual environments inside the mounted shared folder (at some point a “read only system” error is generated, so only your projects files can be shared).

You can unfold the following section in case you want to try and use a workaround that works for VirtualBox up to version 4 but doesn't seem to work for the latest one. Also note that this workaround only works on Linux / MacOS hosts, not on Windows hosts.

Setting up shared folders for a development environment

Before anything else, we'll create a VirtualBox shared folder that will let us easily edit our project files from the host machine. Create a directory on your host that will later be used in the VM as our project's owner user's home directory:

> mkdir ~/Documents/django_home

Then, in the VirtualBox UI, make this directory a permanent shared directory for your Django VM.

Once done, since VirtualBox doesn't initially allow symlinks creation on shared folder, we need to tweak our VM a little bit, read this paragraph for a deeper explanation.

Make sure your VM is not running before issuing the following command:

> VBoxManage setextradata <VM_NAME> VBoxInternal2/SharedFoldersEnableSymlinksCreate/<SHARE_NAME> 1

Check that it is done:

> VBoxManage getextradata <VM_NAME> enumerate

Automatic Mount of Shared Folder

Next we need to have this shared folder automatically mounted as the home directory of our VM user. Launch your VM and log into it as root. Then follow these steps to configure the shared folder as home directory:

Create VM user

> add user django
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
Changing the user information for django
Enter the new value, or press ENTER for the default
	Full Name []: 
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] Y
> id django
uid=1001(django) gid=1001(django) groups=1001(django)

Automount using fstab

First we test-mount our shared folder:

> mount -t vboxsf <sharedfolder-name> </path/to/mountpoint>
> ls -l </path/to/mountpoint>

You should be able to see the content of your mounted shared folder by listing the mountpoint content.

> umount <sharefolder-name>

Once we are sure this works, we automate the mount process at boot time in /etc/fstab:

> nano /etc/fstab

ADD
# automount python-share as root:www-data
<sharedfolder-name>   </path/to/mountpoint>  vboxsf   defaults,uid=1001,gid=33,dmode=775,fmode=664,umask=002  0   0

Now we test that it works:

> mount -a
> ls -la </path/to/mountpoint>

It might be a good idea to reboot the VM to check that there is no problem reported and that the shared folder gets mounted at boot time… Now we can create the top directory for our django project (from your host's desktop environment !)

There are a few other ways to share files between your host system and a VM, like NFS or CIFS/SAMBA, you can learn more about those possibilities by reading this VirtualBox article.

Here we'll focus on one last way of sharing files from our host's filesystem to our VM, using SSHFS.

The following steps are inspired by this DigitalOcean article, the main difference being that we'll access our host directory from our guest VM which is the opposite of what's described in the initial article.

Install SSHFS

Unbuntu/Debian

First check whether SSHFS is already installed on your VM system:

From your VM system

> apt-cache policy sshfs
sshfs:
  Installed: 2.8-1
  Candidate: 2.8-1
  Version table:
     2.8-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

In case you get Installed: (none), then you need to install sshfs as follow:

From your VM system

> sudo apt-get install sshfs

On Mac OSX

You can install SSHFS on Mac OSX. You will need to download FUSE and SSHFS from the osxfuse site

On Windows

To install SSHFS in Windows you will need to grab the latest win-sshfs package from the google code repository. A direct download link can be found below. After you have downloaded the package, double click to launch the installer. You may be prompted to download additional files, if so the installer will download the .NET Framework 4.0 and install it for you.

https://win-sshfs.googlecode.com/files/win-sshfs-0.0.1.5-setup.exe

Mounting the host File System

Create a Django user

As we are willing to set up a Django development environment, we'll first create a user that will be used as our Django reference user:

from the VM console

> adduser django
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
Changing the user information for django
Enter the new value, or press ENTER for the default
	Full Name []: 
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] Y
> id django
uid=1001(django) gid=1001(django) groups=1001(django)

This will create a new /home/django/ directory inside our VM's filesystem. In this directory, we'll create a mount point, /home/django/projects, for our development files residing under our host's directory. Let's say that we have our development files located under the following host's directory: /home/devuser/Documents/dev/django.

Testing SSHFS mount

in your VM's console

> su django
> mkdir /home/django/projects
> exit
> sshfs devuser@172.20.20.1:/home/devuser/Documents/dev/django /home/django/projects -o allow_other,uid=1001,gid=1001
The authenticity of host '172.20.20.1 (172.20.20.1)' can't be established.
ECDSA key fingerprint is SHA256:ck1WSFJeryeGqN86bK3Zci37W8ZrDapOzi/O9aTJo60.
Are you sure you want to continue connecting (yes/no)? yes
devuser@172.20.20.1's password:

Note that you'll have to uncomment user_allow_other in /etc/fuse.conf to be able to use the allow_other option.

Automating mount at VM startup

RSA keys

To have our host directory automatically mounted at each startup of our VM, we'll need a few more steps. As we've just experienced, mounting the directory through SSHFS the way we've just done it, we're asked for a password. Automating the process at startup will require authentication using an RSA key, so we'll first prepare RSA keys authentication between the VM and our host system:

as django user in VM

> cd ~/
> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ddi/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ddi/.ssh/id_rsa.
Your public key has been saved in /home/ddi/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:RmGAqk6LRpYzpQrrurRgK7Z45tt4G3vMKbA2AlhncII ddi@deb9
The key's randomart image is:
+---[RSA 2048]----+
| .   ...o        |
|E o o  . .       |
|   =    .        |
|  o.o  .         |
|.o+o    S        |
|=O.    .         |
|@++o.o .         |
|@=Oooo=          |
|X%=+++           |
+----[SHA256]-----+

Accept default value for key location and leave password blank.

We now copy our user's id_rsa.pub key to our host system:

as django user in VM

> ssh-copy-id devuser@172.20.20.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ddi/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
thibaut@172.20.20.1's password:

Let's make sure mounting through SSHFS works without a password now by specifying usage of the RSA key:

from VM console

> sshfs devuser@172.20.20.1:/home/devuser/Documents/dev/django /home/django/projects -o allow_other,uid=1001,gid=1001,IdentityFile=/home/django/.ssh/id_rsa

Make sure you can access your host directory under the mountpoint and you're all set to finalize the automation.

Ownership

Up until now we simply mounted the directory and all it's content with our django user and group ids (1001). But for Apache to be able to interact with the database we'll need the ability to change the group owner of our project's directory as well as the database file. This is not possible when fixing uid and gid at mount time. As described in the SSHFS man page, we'll use the SSHFS options idmap=file, uidfile and gidfile to map our host ids to ou VM ids.

We'll thus create two files for id mappings and then edit /etc/fstab to set the options to use those files. We are considering the following situation:

On host system:
- user devuser: uid=1000(devuser) gid=1000(devuser)
- user www-data: uid=33(www-data) gid=33(www-data)
On VM system:
- user django: uid=1001(django) gid=1001(django)
- user www-data: uid=33(www-data) gid=33(www-data)

django user in VM console

> mkdir /home/django/sshfs
> nano /home/django/sshfs/sshfs_uids
ADD
django:1000
www-data:33

> nano /home/django/sshfs/sshfs_gids
ADD
django:1000
www-data:33

> sudo nano /etc/fstab

ADD
# Automount development directory from Host using SSHFS
devuser@172.20.20.1:/home/devuser/Documents/dev/django /home/django/projects fuse.sshfs _netdev,allow_other,idmap=file,uidfile=/home/django/sshfs/sshfs_uids,gidfile=/home/django/sshfs/sshfs_gids,nomap=ignore,IdentityFile=/home/django/.ssh/id_rsa,reconnect	0	0

For more details and options, see this ArchLinux page, the SSHFS manual page and the FUSE man page.

Our SSHFS shared directory is ready to be used for development !

Note that you will need to modify ownership properties from the host system.

Using the chown command from your VM console will give a Permission denied error.

Ownership modifications commands will have to be executed from the host console for files accessed through SSHFS, using numeric ids and taking the id mapping into consideration.

From linuxconfig.org

By default Debian 9(.4.0) comes with two versions of Python installed Python 2.7(.13) and Python 3.5(.3), to allow easy switching between the two, we'll use update-alternatives as follow.

First we need to determine what Python binaries are available:

> ls /usr/bin/python*
/usr/bin/python     /usr/bin/python2.7-config  /usr/bin/python3.5   /usr/bin/python-config
/usr/bin/python2    /usr/bin/python2-config    /usr/bin/python3.5m
/usr/bin/python2.7  /usr/bin/python3           /usr/bin/python3m

Next, update the Python alternatives list for each version we whish to use (/usr/bin/python2.7 and /usr/bin/python3.5):

> update-alternatives --install /usr/bin/python python /usr/bin/python3.5 1
update-alternatives: using /usr/bin/python3.5 to provide /usr/bin/python (python) in auto mode
> update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2
update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode

Note that the integer at the end of the command line defines default priority. In this case, we keep python2.7 as the default priority since it has a higher number than python3.5.

From now on, switching to a different version of python is as simple as:

> update-alternatives --config python
There are 2 choices for the alternative python (providing /usr/bin/python).

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /usr/bin/python2.7   2         auto mode
  1            /usr/bin/python2.7   2         manual mode
  2            /usr/bin/python3.5   1         manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/bin/python3.5 to provide /usr/bin/python (python) in manual mode

python --version
Python 3.5.3

We keep python 2.7 as default to make sure nothing brakes as some packages, commands and utilities are relying on it. But we'll be able to use Python 3 for our Django development.

Python 3.6(.4)

In case you need to use Python 3.6, here is how to compile it on Debian 9. As this is quite a power hungry process, one might consider allowing sufficient resources to the VM before executing those commands, with an i7-2720QM CPU, a 4 core / 1 GB RAM VM takes about 20 minutes to complete the compilation…

As root user

> apt-get update && apt-get upgrade
> apt-get install -y make build-essential libssl-dev zlib1g-dev
> apt-get install -y libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
> apt-get install -y libncurses5-dev  libncursesw5-dev xz-utils tk-dev
> wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz
> tar xvf Python-3.6.4.tgz
> cd Python-3.6.4
> ./configure --enable-optimizations
> make -j8
> make altinstall

To test that Python 3.6 is well installed and working, enter a Python shell:

> python3.6
Python 3.6.4 (default, Aug 27 2018, 23:04:48)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

The Python3.6(.4) binary will be located in /usr/local/bin/python3.6
You might want to add it to the available alternatives using the above mentioned method.

Using virtualenv

We now need to create a Python virtual environment so that our Django project will be separated from the system's tools and any other Python projects we may be working on, we need to install the virtualenv command to create these environments, this can be done using the pip command. Lets' install it for Python 3:

As dev user

> update-alternatives --config python
update-alternatives --config python
There are 2 choices for the alternative python (providing /usr/bin/python).

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /usr/bin/python2.7   2         auto mode
  1            /usr/bin/python2.7   2         manual mode
  2            /usr/bin/python3.5   1         manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/bin/python3.5 to provide /usr/bin/python (python) in manual mode

> python --version
Python 3.5.3

> pip3 install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl (1.9MB)
    100% |████████████████████████████████| 1.9MB 665kB/s 
Installing collected packages: virtualenv
Successfully installed virtualenv-16.0.0

Make sure to act as your django user for the next steps !

> su django
> cd </home/django/projects>
> mkdir <my_project>
> cd <my_project>

As we are now in the main project directory, let's create the virtual environment and activate it, making sure we are using Python 3. If the wrong version of python is active, switch to root and execute update-alternatives –config python to activate python 3.:

> python --version
Python 3.5.3
> virtualenv <djangoenv>
> source <djangoenv>/bin/activate
(<djangoenv>) >

Your prompt should change to indicate that you are now operating within a Python virtual environment, indicating the virtualenv name between parenthesis.

Using virtualenvwrapper

virtualenvwrapper keeps all your virtualenvs in one place, and provides convenient tools for activating and deactivating them.

As dev user

> pip3 install virtualenvwrapper
> nano ~/.bashrc

ADD
# LOAD VIRTUALENVWRAPPER AUTOMATICALLY
source virtualenvwrapper.sh

> source ~/.bashrc

Let's create a virtualenv specifying the Python version to use:

As dev user

> mkvirtualenv --python=python3.5 <djangoenv>

To activate the superlist virtualenv:

As dev user

> workon <djangoenv>
(<djangoenv>) >

To deactivate the current virtualenv:

As dev user

(<djangoenv>) > deactivate
>

In case you ever need to remove a virtualenv:

As dev user

> rmvirtualenv <djangoenv>

Now that we reside in our project's directory, and that we have the associated virtualenv running, we're ready to install the Django framework package:

(djangoenv) django@dev > pip install django
Collecting django
  Downloading https://files.pythonhosted.org/packages/56/0e/afdacb47503b805f3ed213fe732bff05254c8befaa034bbada580be8a0ac/Django-2.0.6-py3-none-any.whl (7.1MB)
    100% |████████████████████████████████| 7.1MB 2.6MB/s
Collecting pytz (from django)
  Downloading https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl (510kB)
    100% |████████████████████████████████| 512kB 2.9MB/s
Installing collected packages: pytz, django
Successfully installed django-2.0.6 pytz-2018.4

Django version

It is possible to select the version of Django you'd like to install using the < or = option:

as dev user

> cd ~/your/project/path/
> pip3 install "django<1.12"

Create and configure the Django project

Initiate the project

Staying in our already created project directory, we'll now initiate our Django project, which will create a second level directory containing the actual code. The key to achieving the correct directory structure is to list the parent directory after the project name:

> django-admin.py startproject myproject ~/projects/myproject

Adjust project's settings

First thing, we need to adjust our project's settings:

> nano ~/projects/myproject/settings.py

. . .
ALLOWED_HOSTS = ["172.20.20.10", "127.0.0.1", "127.0.1.1"]
. . .
. . .

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

Complete Initial Project Setup

We will now use the management script to migrate the initial database schema to our SQLite database, create a superuser and collect all static files:

> cd ~/projects/myproject
> ./manage.py makemigrations
No changes detected
> ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK
> ./manage.py createsuperuser
Username (leave blank to use 'ddi'): admin
Email address: tech@tacticz.com
Password:
Password (again):
This password is too short. It must contain at least 8 characters.
This password is too common.
This password is entirely numeric.
Password:
Password (again):
Superuser created successfully.
> ./manage.py collectstatic
....
....
118 static files copied to '/home/ddi/django/ddi_api/static'.

Our basic Django project should now be ready. Let's starting up the default Django server:

> ./manage.py runserver 0.0.0.0:8000

The project should now be accessible by pointing your host's web browser to the address http://172.20.20.10:8000

In order not to always have to launch the Django server, we will use Apache to serve our project's pages. Clients requests will be translated into the WSGI format that the Django application expects using the mod_wsgi module.

Configure VM and host IP Addresses

Our Apache VHosts will be configured using the ServerName directive in our VHost config file. To achieve this, we'll first decide on a unique VBoxNet0 IP address for the VM and associate this to the desired domain name we'll use to access our dev environment.

Let's say we'd like to be able to test our project typing myproject.dev in our host's web browser. We first need to define our unique VM's IP address in the allowed VBoxNet0 VirtualBox network. Let's say we'd like to have our VM responding under 172.20.20.50:

on VM console

> nano /etc/network/interfaces

...
# The vboxnet network interface
allow-hotplug enp0s8
iface enp0s8 inet static
        address 172.20.20.50/24
        
> reboot

on HOST console

> sudo nano /etc/hosts

ADD
# Django-D9Py3A2w virtual hosts
172.20.20.50    myproject.dev

Configure the Project's Virtual Host

As we want to deploy different projects from the same VM, we'll create a new Apache Virtual Host for our project, let's create the configuration file from the default Apache one. We will modify the configuration so that it answers on requests posted on the port 80 with the domain name myproject.dev. Then we'll make sure static files are served from our project's defined static directory. We'll grant access to the wsgi.py file within the second level project directory where the Django code is stored.

Finally we'll construct the directives to handle the WSGI pass, we'll use daemon mode to run the WSGI process, which is the recommended configuration, the WSGIDaemonProcess directive will be used to set this up, for consistency myproject will be used as an arbitrary name for the process. We point the Python home to our virtual environment as this is where Apache can find all of the components that may be required, and the Python path to point to the base of our Django project. We need to specify the process group, this should point to the same name we selected for the WSGIDaemonProcess directive (myproject). We finish by setting the script alias so that Apache will pass requests for the root domain to the wsgi.py file.

> cp -a /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/010-myproject.conf
> nano /etc/apache2/sites-available/010-myproject.conf

CHANGE
#ServerName www.example.com
TO
ServerName myproject.dev

ADD
    # Grant access to static files
    Alias /static /home/django/projects/myproject/static
    <Directory /home/django/projects/myproject/static>
        Require all granted
    </Directory>

    # Grant access to uwsgi.py file
    <Directory /home/django/projects/myproject/myproject>
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>

    # Handle the WSGI pass
    WSGIDaemonProcess myproject python-home=/home/django/projects/myproject/myprojectenv python-path=/home/django/projects/myproject
    WSGIProcessGroup myproject
    WSGIScriptAlias / /home/django/projects/myproject/myproject/wsgi.py

Grant Accesses

We still need to grant write accesses so that Apache can access the database file as well as our project directory:

> chmod 664 /home/django/projects/myproject/db.sqlite3
> chmod 775 /home/django/projects/myproject

We then need to grant ownership of those files to the www-data group which Apache runs under:

> sudo chown :www-data /home/django/projects/myproject/db.sqlite3
> sudo chown :www-data /home/django/projects/myproject

Those chown operations will return a Permission denied warning on the VM side when using SSHFS. As explained earlier, ownership modifications for those files will have to be executed from the host console, using numeric ids and taking the id mapping into consideration.

Restart Apache Server

To make sure we made no mistake writing our configuration file, we'll first activate the new config file and check the syntax, then restart the server:

> sudo a2ensite 010-myproject
To activate the new configuration, you need to run:
  systemctl reload apache2
> sudo apache2ctl configtest
...
Syntax OK
> systemctl reload apache2

We should now be able to access our dev environment typing by myproject.dev in our host's web browser.

If you plan to use Selenium for functional tests or to scrape websites content, you'll need to install Firefox and Geckodriver.

Though Firefox can be launched in headless mode using the -headless option, it still requires the libgtk-3-0 and xvfb packages to be installed in order to run, this has been reported in Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=1372998) but seems unlikely to ever be addressed by the Mozilla community

Here are the steps to get Firefox running on a headless (no X11) system:

As root

> wget -O FirefoxSetup.tar.bz2 "https://download.mozilla.org/?product=firefox-latest&os=linux64&lang=en-US"
> tar xvf FirefoxSetup.tar.bz2
> mv firefox/ /opt/
> apt-get install libgtk-3-0 xvfb
> /opt/firefox/firefox -headless
*** You are running in headless mode.

You can check for the latest Geckodriver version on this github page. Note that we will install the geckodriver binary in our local user path, so we'll update our user's .bashrc file to access it under ~/.local/bin, because this is also where Python will install things when you use pip install –user.

As dev user

> mkdir -p ~/.local/bin
> cd ~/.local/bin
> wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar.gz
> tar xvf geckodriver-v0.21.0-linux64.tar.gz
> nano ~/.bashrc

ADD
# LOCAL BINARIES
PATH=~/.local/bin/:$PATH

> geckodriver --version
geckodriver 0.21.0
...
...

Django 2 with Apache wsgi

Required Additional Packages

Shared Folder Configuration

Automatic Mount of Shared Folder

Create VM user

Automount using fstab

Using SSHFS to access your development files

Install SSHFS

Unbuntu/Debian

On Mac OSX

On Windows

Mounting the host File System

Create a Django user

Testing SSHFS mount

Automating mount at VM startup

Alternative Python versions

Python 3.6(.4)

Configure the Python Virtual Environment

Using virtualenv

Using virtualenvwrapper

Install Django

Django version

Create and configure the Django project

Initiate the project

Adjust project's settings

Complete Initial Project Setup

Use Apache with WSGI

Configure VM and host IP Addresses

Configure the Project's Virtual Host

Grant Accesses

Restart Apache Server

Selenium, Firefox & Geckodriver