====== Django 2 with Apache wsgi ====== From [[https://www.digitalocean.com/community/tutorials/how-to-serve-django-applications-with-apache-and-mod_wsgi-on-debian-8| this DigitalOcean tutorial]]. ===== Required Additional Packages ===== To complete our system setup we will require 3 additional packages, as we intend to use Django 2+, we'll install the required packages for python 3, let's install them: > apt-get update > apt-get install python3-pip apache2 libapache2-mod-wsgi-py3 We need to identify the apache runtime user id and guid for later mounting our shared folder: > id www-data uid=33(www-data) gid=33(www-data) groups=33(www-data) ===== Shared Folder Configuration ===== VirtualBox has **chosen** not to allow symlinks to work in shared folders, you can learn more on the subject by reading [[https://www.virtualbox.org/ticket/10085|this virtualbox ticket]]. This is obviously a **major problem** as this will prevent you from deploying python virtual environments inside the mounted shared folder (at some point a “read only system” error is generated, so only your projects files can be shared). You can unfold the following section in case you want to try and use a workaround that works for VirtualBox up to version 4 but doesn't seem to work for the latest one. Also note that this workaround only works on Linux / MacOS hosts, not on Windows hosts. ++++Setting up shared folders for a development environment| Before anything else, we'll create a VirtualBox shared folder that will let us easily edit our project files from the host machine. Create a directory on your host that will later be used in the VM as our project's owner user's home directory: > mkdir ~/Documents/django_home Then, in the VirtualBox UI, make this directory a permanent shared directory for your Django VM. Once done, since VirtualBox doesn't initially allow symlinks creation on shared folder, we need to tweak our VM a little bit, read [[http://wiki.strategicz.com/vhyper/doku.php?id=vms:python:django#virtualbox_symlinks_inside_shared_folders| this paragraph for a deeper explanation]]. Make sure your VM is **not** running before issuing the following command: > VBoxManage setextradata VBoxInternal2/SharedFoldersEnableSymlinksCreate/ 1 Check that it is done: > VBoxManage getextradata enumerate ==== Automatic Mount of Shared Folder ==== Next we need to have this shared folder automatically mounted as the home directory of our VM user. Launch your VM and log into it as root. Then follow these steps to configure the shared folder as home directory: === Create VM user === > add user django Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully Changing the user information for django Enter the new value, or press ENTER for the default Full Name []: Room Number []: Work Phone []: Home Phone []: Other []: Is the information correct? [Y/n] Y > id django uid=1001(django) gid=1001(django) groups=1001(django) === Automount using fstab === First we test-mount our shared folder: > mount -t vboxsf > ls -l You should be able to see the content of your mounted shared folder by listing the mountpoint content. > umount Once we are sure this works, we automate the mount process at boot time in ''/etc/fstab'': > nano /etc/fstab ADD # automount python-share as root:www-data vboxsf defaults,uid=1001,gid=33,dmode=775,fmode=664,umask=002 0 0 Now we test that it works: > mount -a > ls -la It might be a good idea to reboot the VM to check that there is no problem reported and that the shared folder gets mounted at boot time... Now we can **create the top directory for our django project** (from your host's desktop environment !) ++++ There are a few other ways to share files between your host system and a VM, like **NFS** or **CIFS/SAMBA**, you can learn more about those possibilities by reading [[https://www.virtualbox.org/wiki/Sharing_files_on_OSE|this VirtualBox article]]. ===== Using SSHFS to access your development files ===== Here we'll focus on one last way of sharing files from our host's filesystem to our VM, using **SSHFS**. The following steps are inspired by this [[https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh|DigitalOcean article]], the main difference being that we'll access our //host directory from our guest VM// which is the opposite of what's described in the initial article. ==== Install SSHFS ==== === Unbuntu/Debian === First check whether SSHFS is already installed on your VM system: > apt-cache policy sshfs sshfs: Installed: 2.8-1 Candidate: 2.8-1 Version table: 2.8-1 500 500 http://deb.debian.org/debian stretch/main amd64 Packages 100 /var/lib/dpkg/status In case you get ''Installed: (none)'', then you need to install sshfs as follow: > sudo apt-get install sshfs === On Mac OSX === You can install SSHFS on Mac OSX. You will need to download FUSE and SSHFS from the [[http://osxfuse.github.io/|osxfuse site]] === On Windows === To install SSHFS in Windows you will need to grab the latest win-sshfs package from the google code repository. A direct download link can be found below. After you have downloaded the package, double click to launch the installer. You may be prompted to download additional files, if so the installer will download the .NET Framework 4.0 and install it for you. https://win-sshfs.googlecode.com/files/win-sshfs-0.0.1.5-setup.exe ==== Mounting the host File System ==== === Create a Django user === As we are willing to set up a Django development environment, we'll first create a user that will be used as our Django reference user: > adduser django Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully Changing the user information for django Enter the new value, or press ENTER for the default Full Name []: Room Number []: Work Phone []: Home Phone []: Other []: Is the information correct? [Y/n] Y > id django uid=1001(django) gid=1001(django) groups=1001(django) This will create a new ''/home/django/'' directory inside our VM's filesystem. In this directory, we'll create a mount point, ''/home/django/projects'', for our development files residing under our **host's directory**. Let's say that we have our development files located under the following host's directory: ''/home/devuser/Documents/dev/django''. === Testing SSHFS mount === > su django > mkdir /home/django/projects > exit > sshfs devuser@172.20.20.1:/home/devuser/Documents/dev/django /home/django/projects -o allow_other,uid=1001,gid=1001 The authenticity of host '172.20.20.1 (172.20.20.1)' can't be established. ECDSA key fingerprint is SHA256:ck1WSFJeryeGqN86bK3Zci37W8ZrDapOzi/O9aTJo60. Are you sure you want to continue connecting (yes/no)? yes devuser@172.20.20.1's password: Note that you'll have to **uncomment** ''user_allow_other'' in ''/etc/fuse.conf'' to be able to use the ''allow_other'' option. === Automating mount at VM startup === **RSA keys** To have our //host directory// automatically mounted at each startup of our VM, we'll need a few more steps. As we've just experienced, mounting the directory through **SSHFS** the way we've just done it, we're asked for a password. Automating the process at startup will require authentication using an RSA key, so we'll first prepare [[https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2|RSA keys authentication]] between the VM and our host system: > cd ~/ > ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/ddi/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ddi/.ssh/id_rsa. Your public key has been saved in /home/ddi/.ssh/id_rsa.pub. The key fingerprint is: SHA256:RmGAqk6LRpYzpQrrurRgK7Z45tt4G3vMKbA2AlhncII ddi@deb9 The key's randomart image is: +---[RSA 2048]----+ | . ...o | |E o o . . | | = . | | o.o . | |.o+o S | |=O. . | |@++o.o . | |@=Oooo= | |X%=+++ | +----[SHA256]-----+ Accept default value for key location and leave password blank. We now copy our user's ''id_rsa.pub'' key to our host system: > ssh-copy-id devuser@172.20.20.1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ddi/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys thibaut@172.20.20.1's password: Let's make sure mounting through SSHFS works without a password now by specifying usage of the RSA key: > sshfs devuser@172.20.20.1:/home/devuser/Documents/dev/django /home/django/projects -o allow_other,uid=1001,gid=1001,IdentityFile=/home/django/.ssh/id_rsa Make sure you can access your //host directory// under the mountpoint and you're all set to finalize the automation. **Ownership** Up until now we simply mounted the directory and all it's content with our django user and group ids (''1001''). But for Apache to be able to interact with the database we'll need the ability to change the group owner of our project's directory as well as the database file. This is not possible when //fixing// ''uid'' and ''gid'' at mount time. As described in the [[https://jlk.fjfi.cvut.cz/arch/manpages/man/sshfs.1|SSHFS man page]], we'll use the SSHFS options ''idmap=file'', ''uidfile'' and ''gidfile'' to **map our host ids to ou VM ids**. We'll thus create two files for id mappings and then edit ''/etc/fstab'' to set the options to use those files. We are considering the following situation: * **On host system**: * user ''devuser'': ''uid=1000(devuser) gid=1000(devuser)'' * user ''www-data'': ''uid=33(www-data) gid=33(www-data)'' * **On VM system**: * user ''django'': ''uid=1001(django) gid=1001(django)'' * user ''www-data'': ''uid=33(www-data) gid=33(www-data)'' > mkdir /home/django/sshfs > nano /home/django/sshfs/sshfs_uids ADD django:1000 www-data:33 > nano /home/django/sshfs/sshfs_gids ADD django:1000 www-data:33 > sudo nano /etc/fstab ADD # Automount development directory from Host using SSHFS devuser@172.20.20.1:/home/devuser/Documents/dev/django /home/django/projects fuse.sshfs _netdev,allow_other,idmap=file,uidfile=/home/django/sshfs/sshfs_uids,gidfile=/home/django/sshfs/sshfs_gids,nomap=ignore,IdentityFile=/home/django/.ssh/id_rsa,reconnect 0 0 For more details and options, see [[https://wiki.archlinux.org/index.php/SSHFS#Automounting|this ArchLinux page]], the [[https://jlk.fjfi.cvut.cz/arch/manpages/man/sshfs.1|SSHFS manual page]] and the [[http://man7.org/linux/man-pages/man8/mount.fuse.8.html|FUSE man page]]. Our SSHFS shared directory is ready to be used for development ! Note that **you will need to modify ownership properties from the host system**. Using the ''chown'' command from your VM console will give a ''Permission denied'' error. Ownership modifications commands will have to be executed from the host console for files accessed through SSHFS, using numeric ids and taking the id mapping into consideration. ===== Alternative Python versions ===== From [[https://linuxconfig.org/how-to-change-default-python-version-on-debian-9-stretch-linux|linuxconfig.org]] By default Debian 9(.4.0) comes with two versions of Python installed Python 2.7(.13) and Python 3.5(.3), to allow easy switching between the two, we'll use ''update-alternatives'' as follow. First we need to determine what Python binaries are available: > ls /usr/bin/python* /usr/bin/python /usr/bin/python2.7-config /usr/bin/python3.5 /usr/bin/python-config /usr/bin/python2 /usr/bin/python2-config /usr/bin/python3.5m /usr/bin/python2.7 /usr/bin/python3 /usr/bin/python3m Next, update the Python alternatives list for each version we whish to use (/usr/bin/python2.7 and /usr/bin/python3.5): > update-alternatives --install /usr/bin/python python /usr/bin/python3.5 1 update-alternatives: using /usr/bin/python3.5 to provide /usr/bin/python (python) in auto mode > update-alternatives --install /usr/bin/python python /usr/bin/python2.7 2 update-alternatives: using /usr/bin/python2.7 to provide /usr/bin/python (python) in auto mode Note that the integer at the end of the command line defines default priority. In this case, we keep python2.7 as the default priority since it has a higher number than python3.5. From now on, switching to a different version of python is as simple as: > update-alternatives --config python There are 2 choices for the alternative python (providing /usr/bin/python). Selection Path Priority Status ------------------------------------------------------------ * 0 /usr/bin/python2.7 2 auto mode 1 /usr/bin/python2.7 2 manual mode 2 /usr/bin/python3.5 1 manual mode Press to keep the current choice[*], or type selection number: 2 update-alternatives: using /usr/bin/python3.5 to provide /usr/bin/python (python) in manual mode python --version Python 3.5.3 We keep python 2.7 as default to make sure nothing brakes as some packages, commands and utilities are relying on it. But we'll be able to use Python 3 for our Django development. ==== Python 3.6(.4) ==== In case you need to use Python 3.6, here is how to compile it on Debian 9. As this is quite a power hungry process, one might consider allowing sufficient resources to the VM before executing those commands, with an i7-2720QM CPU, a 4 core / 1 GB RAM VM takes about 20 minutes to complete the compilation... > apt-get update && apt-get upgrade > apt-get install -y make build-essential libssl-dev zlib1g-dev > apt-get install -y libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm > apt-get install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev > wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz > tar xvf Python-3.6.4.tgz > cd Python-3.6.4 > ./configure --enable-optimizations > make -j8 > make altinstall To test that Python 3.6 is well installed and working, enter a Python shell: > python3.6 Python 3.6.4 (default, Aug 27 2018, 23:04:48) [GCC 6.3.0 20170516] on linux Type "help", "copyright", "credits" or "license" for more information. >>> The Python3.6(.4) binary will be located in ''/usr/local/bin/python3.6''\\ You might want to add it to the available alternatives using the above mentioned method. ===== Configure the Python Virtual Environment ===== ==== Using virtualenv ==== We now need to create a Python virtual environment so that our Django project will be separated from the system's tools and any other Python projects we may be working on, we need to install the ''virtualenv'' command to create these environments, this can be done using the ''pip'' command. Lets' install it **for Python 3**: > update-alternatives --config python update-alternatives --config python There are 2 choices for the alternative python (providing /usr/bin/python). Selection Path Priority Status ------------------------------------------------------------ * 0 /usr/bin/python2.7 2 auto mode 1 /usr/bin/python2.7 2 manual mode 2 /usr/bin/python3.5 1 manual mode Press to keep the current choice[*], or type selection number: 2 update-alternatives: using /usr/bin/python3.5 to provide /usr/bin/python (python) in manual mode > python --version Python 3.5.3 > pip3 install virtualenv Collecting virtualenv Downloading https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl (1.9MB) 100% |████████████████████████████████| 1.9MB 665kB/s Installing collected packages: virtualenv Successfully installed virtualenv-16.0.0 Make sure to **act as your django user** for the next steps ! > su django > cd > mkdir > cd As we are now in the main project directory, let's create the virtual environment and activate it, **making sure we are using Python 3**. If the wrong version of python is active, switch to root and execute ''update-alternatives --config python'' to activate python 3.: > python --version Python 3.5.3 > virtualenv > source /bin/activate () > Your prompt should change to indicate that you are now operating within a Python virtual environment, indicating the virtualenv name between parenthesis. ==== Using virtualenvwrapper ==== ''virtualenvwrapper'' keeps all your virtualenvs in one place, and provides convenient tools for activating and deactivating them. > pip3 install virtualenvwrapper > nano ~/.bashrc ADD # LOAD VIRTUALENVWRAPPER AUTOMATICALLY source virtualenvwrapper.sh > source ~/.bashrc Let's create a ''virtualenv'' specifying the Python version to use: > mkvirtualenv --python=python3.5 To **activate** the ''superlist virtualenv'': > workon () > To **deactivate** the current ''virtualenv'': () > deactivate > In case you ever need to **remove** a ''virtualenv'': > rmvirtualenv ===== Install Django ===== Now that we reside in our project's directory, and that we have the associated virtualenv running, we're ready to install the Django framework package: (djangoenv) django@dev > pip install django Collecting django Downloading https://files.pythonhosted.org/packages/56/0e/afdacb47503b805f3ed213fe732bff05254c8befaa034bbada580be8a0ac/Django-2.0.6-py3-none-any.whl (7.1MB) 100% |████████████████████████████████| 7.1MB 2.6MB/s Collecting pytz (from django) Downloading https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl (510kB) 100% |████████████████████████████████| 512kB 2.9MB/s Installing collected packages: pytz, django Successfully installed django-2.0.6 pytz-2018.4 === Django version === It is possible to select the version of Django you'd like to install using the ''<'' or ''='' option: > cd ~/your/project/path/ > pip3 install "django<1.12" ==== Create and configure the Django project ==== === Initiate the project === Staying in our already created project directory, we'll now initiate our Django project, which will create a second level directory containing the actual code. **The key** to achieving the correct directory structure is to **list the parent directory after the project name**: > django-admin.py startproject myproject ~/projects/myproject === Adjust project's settings === First thing, we need to adjust our project's settings: > nano ~/projects/myproject/settings.py . . . ALLOWED_HOSTS = ["172.20.20.10", "127.0.0.1", "127.0.1.1"] . . . . . . STATIC_URL = '/static/' STATIC_ROOT = os.path.join(BASE_DIR, 'static/') ==== Complete Initial Project Setup ==== We will now use the management script to migrate the initial database schema to our SQLite database, create a superuser and collect all static files: > cd ~/projects/myproject > ./manage.py makemigrations No changes detected > ./manage.py migrate Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying sessions.0001_initial... OK > ./manage.py createsuperuser Username (leave blank to use 'ddi'): admin Email address: tech@tacticz.com Password: Password (again): This password is too short. It must contain at least 8 characters. This password is too common. This password is entirely numeric. Password: Password (again): Superuser created successfully. > ./manage.py collectstatic .... .... 118 static files copied to '/home/ddi/django/ddi_api/static'. Our basic Django project should now be ready. Let's starting up the default Django server: > ./manage.py runserver 0.0.0.0:8000 The project should now be accessible by pointing your host's web browser to the address ''http://172.20.20.10:8000'' ===== Use Apache with WSGI ===== In order not to always have to launch the Django server, we will use Apache to serve our project's pages. Clients requests will be translated into the WSGI format that the Django application expects using the ''mod_wsgi'' module. ==== Configure VM and host IP Addresses ==== Our Apache VHosts will be configured using the ''ServerName'' directive in our VHost config file. To achieve this, we'll first decide on a unique VBoxNet0 IP address for the VM and associate this to the desired domain name we'll use to access our dev environment. Let's say we'd like to be able to test our project typing ''myproject.dev'' in our host's web browser. We first need to define our unique VM's IP address in the allowed VBoxNet0 VirtualBox network. Let's say we'd like to have our VM responding under ''172.20.20.50'': > nano /etc/network/interfaces ... # The vboxnet network interface allow-hotplug enp0s8 iface enp0s8 inet static address 172.20.20.50/24 > reboot > sudo nano /etc/hosts ADD # Django-D9Py3A2w virtual hosts 172.20.20.50 myproject.dev ==== Configure the Project's Virtual Host ==== As we want to deploy different projects from the same VM, we'll create a new Apache Virtual Host for our project, let's create the configuration file from the default Apache one. We will modify the configuration so that it answers on requests posted on the port 80 with the domain name ''myproject.dev''. Then we'll make sure static files are served from our project's defined ''static'' directory. We'll grant access to the ''wsgi.py'' file within the second level project directory where the Django code is stored. Finally we'll construct the directives to handle the WSGI pass, we'll use daemon mode to run the WSGI process, which is the recommended configuration, the ''WSGIDaemonProcess'' directive will be used to set this up, for consistency ''myproject'' will be used as an arbitrary name for the process. We point the Python home to our virtual environment as this is where Apache can find all of the components that may be required, and the Python path to point to the base of our Django project. We need to specify the process group, this should point to the same name we selected for the ''WSGIDaemonProcess'' directive (''myproject''). We finish by setting the script alias so that Apache will pass requests for the root domain to the ''wsgi.py'' file. > cp -a /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/010-myproject.conf > nano /etc/apache2/sites-available/010-myproject.conf CHANGE #ServerName www.example.com TO ServerName myproject.dev ADD # Grant access to static files Alias /static /home/django/projects/myproject/static Require all granted # Grant access to uwsgi.py file Require all granted # Handle the WSGI pass WSGIDaemonProcess myproject python-home=/home/django/projects/myproject/myprojectenv python-path=/home/django/projects/myproject WSGIProcessGroup myproject WSGIScriptAlias / /home/django/projects/myproject/myproject/wsgi.py ==== Grant Accesses ==== We still need to grant write accesses so that Apache can access the database file as well as our project directory: > chmod 664 /home/django/projects/myproject/db.sqlite3 > chmod 775 /home/django/projects/myproject We then need to grant ownership of those files to the ''www-data'' group which Apache runs under: > sudo chown :www-data /home/django/projects/myproject/db.sqlite3 > sudo chown :www-data /home/django/projects/myproject Those ''chown'' operations will return a ''Permission denied'' warning on the VM side when using ''SSHFS''. As explained earlier, ownership modifications for those files will have to be executed from the host console, using numeric ids and taking the id mapping into consideration. ==== Restart Apache Server ==== To make sure we made no mistake writing our configuration file, we'll first activate the new config file and check the syntax, then restart the server: > sudo a2ensite 010-myproject To activate the new configuration, you need to run: systemctl reload apache2 > sudo apache2ctl configtest ... Syntax OK > systemctl reload apache2 We should now be able to access our dev environment typing by ''myproject.dev'' in our host's web browser. ===== Selenium, Firefox & Geckodriver ===== If you plan to use Selenium for functional tests or to scrape websites content, you'll need to install Firefox and Geckodriver. Though Firefox can be launched in //headless// mode using the ''-headless'' option, it still requires the ''libgtk-3-0'' and ''xvfb'' packages to be installed in order to run, this has been reported in Bugzilla ([[https://bugzilla.mozilla.org/show_bug.cgi?id=1372998]]) but seems unlikely to ever be addressed by the Mozilla community :-( Here are the steps to get Firefox running on a headless (no X11) system: > wget -O FirefoxSetup.tar.bz2 "https://download.mozilla.org/?product=firefox-latest&os=linux64&lang=en-US" > tar xvf FirefoxSetup.tar.bz2 > mv firefox/ /opt/ > apt-get install libgtk-3-0 xvfb > /opt/firefox/firefox -headless *** You are running in headless mode. You can check for the latest Geckodriver version on [[https://github.com/mozilla/geckodriver/releases|this github page]]. Note that we will install the ''geckodriver'' binary in our local user path, so we'll update our user's ''.bashrc'' file to access it under ''~/.local/bin'', because this is also where Python will install things when you use ''pip install --user''. > mkdir -p ~/.local/bin > cd ~/.local/bin > wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar.gz > tar xvf geckodriver-v0.21.0-linux64.tar.gz > nano ~/.bashrc ADD # LOCAL BINARIES PATH=~/.local/bin/:$PATH > geckodriver --version geckodriver 0.21.0 ... ...