Python - uWSGI - Django - nginx as Development Environment

Turnkey Linux VM (deprecated)

We'll start with a Debian template.
Then:

> sudo apt-get update
> apt-get upgrade

using two bridged interfaces. One using DHCP for Internet access and the other using a fixed IP for management and test access.

> ifconfig -a
> nano /etc/network/interfaces

ADD:
# The admin network interface
auto <eth1>
iface <eth1> inet static
   address 172.20.20.1
   netmask 255.255.255.0
   broadcast 172.20.20.255
   network 172.20.20.0

> ifdown <eth1> && ifup <eth1>

> nano /etc/hostname
> nano /etc/hosts

> nano /etc/ssh/sshd_config
CHANGE FROM:
PermitRootLogin without-password
TO:
PermitRootLogin yes

> service ssh restart

(From the host system)

> ssh-copy-id -i root@<host.ip.address>

You might want to reboot the VM at this point and login through a terminal instead of using the VirtualBox restricted screen.

As root:

> apt-get install sudo

Add sysadmin user to sudo group

> adduser sysadmin sudo

Adding to the sudoers group:

> sudo usermod -a -G sudo <username>

Launch your VBox VM from the VM VirtualBox Manager's main interface.

From the VM's window, select menu Devices > Insert Guest Additions CD image…
Then mount the CD:

> mount /dev/cdrom /media/cdrom
mount:/dev/sr0 is write-protected, mounting read-only

Install necessary packages:

> sudo apt-get install linux-headers-$(uname -r)
> sudo apt-get install build-essential linux-headers-`uname -r` dkms
> sudo /media/cdrom/VBoxLinuxAdditions.run

> sudo apt-get install ranger htop iftop iotop

The following section has been judge overkill, although you're able to read it by

clicking here:

> sudo cp -R /etc/skel/ /etc/skel.bak
> sudo mkdir /etc/skel/.bashrc.d

> sudo useradd -m -g users -G www-data,sudo -s /bin/bash webdev
> sudo passwd webdev
Enter new UNIX password: 
Retype new UNIX password: 

Refer to this answer on serverfault to see how to setup permissions on the /var/python directory that will be used to store our scripts.

You have to create a shared folder for your VM, then “prepare” it for correct symlinks operations from your host computer command line: See Create VirtualBox shared folder in this wiki.

Test-mount your shared folder:

> mount -t vboxsf <sharedfolder-name> </path/to/montpoint>

Check your created (webdev) user uid and gid with:

> id webdev
uid=1001(webdev) gid=100(users) groups=100(users),27(sudo),33(www-data)

Then create your python project directory and make it the target for the shared folder that we'll auto-mount at boot time by write to /etc/fstab as follow:

> sudo mkdir /var/python
> sudo nano /etc/fstab

# automount python-share as root:www-data
python-share   /var/python   vboxsf   defaults,uid=1001,gid=33,dmode=775,fmode=664,umask=002  0   0

> sudo mount -a

USING THE /etc/fstab AUTOMOUNT CAUSES THE DEBIAN 8 STARTUP TO DROP INTO EMERGENCY MODE

There is an, apparently intentional, limitation in VirtualBox that prevents creation of symlinks from the VM inside a shared folder. This seems specifically complex to resolve using a Windows host, you are invited to read the mentioned ticket for more details.

Although on Linux and OS X hosts, using the following command should prevent you from experimenting the dreadful Read-only file system error while creating symlinks inside a shared folder on your guest VM.

> VBoxManage setextradata <VM_NAME> VBoxInternal2/SharedFoldersEnableSymlinksCreate/<SHARE_NAME> 1

This command has to be issued from your host OS, replacing <VM_NAME> with the name of your VirtualBox virtual machine and <SHARE_NAME> with this machine's shared folder name.

You can verify that the command has been taken into account examining the output of:

> VBoxManage getextradata <VM_NAME> enumerate
...
Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/<SHARE_NAME>, Value: 1
...

Not doing so will not let you install your Python virtual environments inside the shared folder. In case you cannot reach a solution (under Windows probably), it is always possible for you to have your virtual environments stored in a directory inside your VM that is NOT a shared folder.

We'll need to run python scripts from our web interface, there are a couple of solutions to do this, but since FreeNAS is using Django, let's go for this one also… we'll hopefully learn something about FreeNAS along the way.

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. We'll use uWSGI as web server gateway interface, following this guide...
We will set up uWSGI so that it creates a Unix socket, and serves responses to the web server via the WSGI protocol.

Python 2.7(.10) is currently the default Python version that comes “pre-installed” with Debian Jessie (8). In case you're willing to use Python 3, here is the summary of the commands to execute, you can refer to the reference article for more details.

First let's make sure what Python version is default on the system and what versions are available:

> python --version
Python 2.7.10
> ls -l /usr/bin/python*
lrwxrwxrwx 1 root root      24 Aug  7 16:12 /usr/bin/python -> python2.7
lrwxrwxrwx 1 root root       9 Mar 16 23:37 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3810568 Jul  1 14:37 /usr/bin/python2.7

Python 3 is NOT available on this system, let's install it, then make it default (using update-alternatives):

> update-alternatives --list python
update-alternatives: error: no alternatives for python

> update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1
> update-alternatives --install /usr/bin/python python /usr/bin/python3.4 2

> update-alternatives --list python
/usr/bin/python2.7
/usr/bin/python3.4
> python --version
Python 3.4.3+

We now have Python 3(.4.3) as our system's default.

A quick intro about Python virtual environments can be found here (in french).

setuptools is a Python code distribution tool, allowing, amongst other, automatic external libs installation and download from the Internet. It's like gem install under Ruby or npm for NodeJS. Python has its installation command called easy_install, picking in the pypi.python.org web site, like Cpan for Perl or Pear channels for PHP.

Apparently pip does the same, but better.

NOTE: It seems that on most distribution, if you're running Python 2.7.9+ or Python 3.4+, pip should already be available. This is not the case in Debian.

With Python 2:

> sudo apt-get install python-pip python-dev build-essential

When using python 3:

> sudo apt-get install python3-pip python3-dev build-essential

Install virtualenv:

> pip install virtualenv
OR
> pip3 install virtualenv

> mkdir -p /var/python/virtenvs
> cd /var/python/virtenvs
> virtualenv fengui

Activate the Python virtual environment:

> source /var/python/virtenvs/fengui/bin/activate
(fengui)webdev@djanginx:>

Notice the virtualenv name in parenthesis in front of the prompt.

To exit the Python virtual environment:

(fengui)> deactivate

Install Django into your virtualenv, create a new project, and cd into the project:

(fengui)> pip install Django
(fengui)> cd /var/python
(fengui)> django-admin.py startproject fengui
(fengui)> cd fengui

To start a new app, after creating the project with Django, enter your project's directory and use:

(fengui)> cd /var/python/fengui
(fengui)> python manage.py startapp <appname>

Inside your virtual environment:

(fengui)> pip install uwsgi
*** uWSGI is ready, launch it with /var/python/virtenv/fengui/bin/uwsgi ***

Create a file called test.py

(fengui)> cd /var/python/fengui
(fengui)> nano test.py

# test.py
def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    #return [b"Hello World"] # python3
    return ["Hello World"] # python2

(fengui)> uwsgi --http :8000 --wsgi-file test.py

Then go to your VM IP address with a web browser: http://172.20.20.4:8000
You should see the “Hello World” page!

This means we now have the following stack of components working:

the web client ↔ uWSGI ↔ Python

Now we want uWSGI to do the same thing, but to run a Django site instead of the test.py module.

If you haven’t already done so, make sure that your gui project actually works:

> cd /var/python/fengui
> python manage.py runserver 172.20.20.4:8000

Then browse http://172.20.20.4:8000, you should see a message like:
It worked!
Congratulations on your first Django-powered page.

And if that works, run it using uWSGI:

> uwsgi --http :8000 --module fengui.wsgi

module gui.wsgi: load the specified wsgi module

Point your browser at the server; if the site appears, it means uWSGI is able to serve your Django application from your virtualenv, and this stack operates correctly:

the web client ↔ uWSGI ↔ Django

Now normally we won’t have the browser speaking directly to uWSGI. That’s a job for the webserver, which will act as a go-between.

Install and start nginx:

> sudo apt-get install nginx
> sudo service nginx start

Browse to http://172.20.20.4, you should see a message like:
Welcome to nginx!

These components of the full stack are working together:

the web client ↔ the web server

You will need the uwsgi_params file, which is available in the nginx directory of the uWSGI distribution, or from https://github.com/nginx/nginx/blob/master/conf/uwsgi_params

Copy it into your project directory. In a moment we will tell nginx to refer to it.

> cd /var/python/fengui
> wget https://raw.githubusercontent.com/nginx/nginx/master/conf/uwsgi_params

The uwsgi_params file contains this:

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;

Now create a file called fengui_nginx.conf, and put this in it:

# fengui_nginx.conf

# the upstream component nginx needs to connect to
upstream django {
    # server unix:///path/to/your/mysite/mysite.sock; # for a file socket
    server 127.0.0.1:8000; # for a web port socket (we'll use this first)
}

# configuration of the server
server {
    # the port your site will be served on
    listen      80;
    # the domain name it will serve for
    server_name 10.0.7.212; # substitute your machine's IP address or FQDN
    charset     utf-8;

    # max upload size
    client_max_body_size 75M;   # adjust to taste

    # Django media
    location /media  {
        alias /var/python/fengui/media;  # your Django project's media files - amend as r$
    }

    location /static {
        alias /var/python/fengui/static; # your Django project's static files - amend as $
    }

    # Finally, send all non-media requests to the Django server.
    location / {
        uwsgi_pass  django;
        include     /var/python/fengui/uwsgi_params; # the uwsgi_params file you install$
    }
}

This conf file tells nginx to serve up media and static files from the filesystem, as well as handle requests that require Django’s intervention. For a large deployment it is considered good practice to let one server handle static/media files, and another handle Django applications, but for now, this will do just fine.

Symlink to this file from /etc/nginx/sites-enabled so nginx can see it:

> sudo ln -s /var/python/fengui/fengui_nginx.conf /etc/nginx/sites-enabled/

Before running nginx, you have to collect all Django static files in the static folder. First of all you have to edit /var/python/freenas/gui/gui/settings.py adding:

> nano /var/python/fengui/fengui/settings.py

ADD:
STATIC_ROOT = os.path.join(BASE_DIR, "static/")

and then run

> python manage.py collectstatic

If you get an error like:
ImportError: No module named django.core.management
Make sure your virtualenv is activated. You can always see it between parenthesis at the start of the prompt line. Or you can issue the following command:

> which python
/usr/bin/python

In this case it's the system installed python that gets executed as your virtualenv is disactivated, go to your virtualenv root and type:

> cd /var/python/fengui
> source bin/activate
(freenas)> which python
/var/python/fengui/bin/python

To check that media files are being served correctly, add an image called media.png to the /var/python/fengui/media directory, then visit http://your.ip.here:8000/media/media.png - if this works, you’ll know at least that nginx is serving files correctly.

It is worth not just restarting nginx, but actually stopping and then starting it again, which will inform you if there is a problem, and where it is.

> mkdir /var/python/fengui/media
> cd /var/python/fengui/media
> wget http://wiki.strategicz.com/vhyper/lib/exe/fetch.php?media=wiki:logo.png
> mv fetch.php?media=wiki:logo.png media.png
> service nginx stop
> service nginx start

Point your browser to http://172.20.20.4:8000/media/media.png, you’ll know at least that nginx is serving files correctly.

Let’s get nginx to speak to the “hello world” test.py application.

(fengui)> cd /var/python/fengui
(fengui)> uwsgi --socket :8000 --wsgi-file test.py

This is nearly the same as before, except this time one of the options is different: socket :8000: use protocol uwsgi, port 8000

nginx meanwhile has been configured to communicate with uWSGI on that port, and with the outside world on port 80. Visit: http://172.20.20.4:80/ to check. And this is our stack:

the web client ↔ the web server ↔ the socket ↔ uWSGI ↔ Python

Meanwhile, you can try to have a look at the uswgi output at http://172.20.20.4:8000 - but quite probably, it won’t work because your browser speaks http, not uWSGI, though you should see output from uWSGI in your terminal.

If you get an error like:
no python application found, check your startup logs for errors
in the command line and:
Internal Server Error
Make sure you issued the uwsgi command from the root of your project.

So far we have used a TCP port socket, because it’s simpler, but in fact it’s better to use Unix sockets than ports - there’s less overhead.

Edit /var/python/projects/fengui/fengui_nginx.conf, changing it to match:

server unix:///var/python/fengui/fengui.sock; # for a file socket
# server 127.0.0.1:8001; # for a web port socket (we'll use this first)

Then restart nginx.

> service nginx restart

Run uWSGI again:

> uwsgi --socket fengui.sock --wsgi-file test.py

This time the socket option tells uWSGI which file to use.

Try http://example.com:8000/ in the browser.

Check your nginx logs at /var/log/nginx/error.log

If you see something like:

...connect() to unix:///var/python/freenas/gui/fengui.sock failed (13: Permission denied) while connecting to upstream...

Try (very permissive):

> uwsgi --socket fegui.sock --wsgi-file test.py --chmod-socket=666

or (more sensible):

> uwsgi --socket fengui.sock --wsgi-file test.py --chmod-socket=664

You may also have to add your user to nginx’s group (which is probably www-data), or vice-versa, so that nginx can read and write to your socket properly.

Let’s run our Django application through a file socket using uwsgi:

(virt)> uwsgi --socket fengui.sock --module fengui.wsgi --chmod-socket=664

OR

(virt)> uwsgi --socket :8001 --module fengui.wsgi

Now uWSGI and nginx should be serving up not just a “Hello World” module, but your Django project.

The working stack is now:

the web client ↔ the web server ↔ the socket ↔ uWSGI ↔ Django

In case you have some scripts that require a long time to execute, you might want to change the uwsgi_read_timeout value in your nginx configuration file, in our above example this would be:

> nano /var/python/fengui/fengui_nginx.conf

...
location / {
        uwsgi_pass         django_pub;
        include            /var/python/fengui/uwsgi_params;
        uwsgi_read_timeout 180s;
    }
...

You can read more about the various nginx directives ruling the uwsgi module on this nginx website's page.

Also you could have rules applied on specific locations:

...
location /volumes {
        uwsgi_pass         django_pub;
        include            /var/python/fengui/uwsgi_params;
        uwsgi_read_timeout 180s;
    }
...

Consult the original article http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html#configuring-uwsgi-to-run-with-a-ini-file

https://docs.djangoproject.com/en/1.6/intro/tutorial01/