====== Python - uWSGI - Django - nginx as Development Environment ====== ----- ++++ Turnkey Linux VM (deprecated)| ===== Turnkey Linux VM ===== Time is running out now, so we do not have much left to get lost in the intricacies of installing a custom nginx server. So let's go for the [[http://www.turnkeylinux.org/nginx-php-fastcgi|Turnkey appliance of nginx]]. We'll download the [[http://www.turnkeylinux.org/download?file=turnkey-nginx-php-fastcgi-13.0-wheezy-amd64-vmdk.zip|nginx php fastcgi 13.0 wheezy amd64 vmdk appliance]] and use it as a development VM using [[https://www.virtualbox.org/|VirtualBox]] on our development machine. * Create a Debian VM, with 2 bridged adapters * Attach the downloaded (and extracted) vmdk * Launch the VM * Set root & mysql passwords * Skip Hub Services * Install security updates * Reach the web interface using the displayed IP address * Select Webmin (accept security warning) * Login as: root + password * Set "System Time": Timezone / Time server sync (be.pool.ntp.org, fr.pool.ntp.org, europe.pool.ntp.org) * Select "Networking" menu, then "Network Configuration", then "Network Interfaces" * Leave eth0 to DHCP (for Internet connection) * Set eth1 to static local admin IP address: 172.20.20.4 / 255.255.255.0 * ssh as root into the console using the dhcp allocated address, then: ifdown eth1 && ifup eth1 * Try to ping 172.20.20.4 from your host system ++++ ----- ===== Debian VM ===== We'll start with a [[..:debian|Debian template]].\\ Then: > sudo apt-get update > apt-get upgrade ----- ---- ==== Setup Network Interfaces ==== using two bridged interfaces. One using DHCP for Internet access and the other using a fixed IP for management and test access. === Activate the Dev Network NIC interface === > ifconfig -a > nano /etc/network/interfaces ADD: # The admin network interface auto iface inet static address 172.20.20.1 netmask 255.255.255.0 broadcast 172.20.20.255 network 172.20.20.0 > ifdown && ifup === Set hostname === > nano /etc/hostname > nano /etc/hosts ---- ==== Allow ssh root login (debian 8) ==== > nano /etc/ssh/sshd_config CHANGE FROM: PermitRootLogin without-password TO: PermitRootLogin yes > service ssh restart === Put your public rsa key for login === (From the host system) > ssh-copy-id -i root@ You might want to reboot the VM at this point and login through a terminal instead of using the VirtualBox restricted screen. === Install sudo === As root: > apt-get install sudo Add sysadmin user to sudo group > adduser sysadmin sudo Adding to the sudoers group: > sudo usermod -a -G sudo ---- ==== Install VirtualBox Guest Additions ==== Launch your VBox VM from the VM VirtualBox Manager's main interface. From the VM's window, select menu **Devices > Insert Guest Additions CD image...**\\ Then mount the CD: > mount /dev/cdrom /media/cdrom mount:/dev/sr0 is write-protected, mounting read-only Install necessary packages: > sudo apt-get install linux-headers-$(uname -r) > sudo apt-get install build-essential linux-headers-`uname -r` dkms > sudo /media/cdrom/VBoxLinuxAdditions.run ---- ==== Add some useful packages ==== > sudo apt-get install ranger htop iftop iotop ---- ==== Configure default bash profile ==== The following section has been judge overkill, although you're able to read it by ++++ clicking here:| Set-up default .bashrc: > sudo cp -R /etc/skel/ /etc/skel.bak > sudo mkdir /etc/skel/.bashrc.d Edit .bashrc and .bashrc.d/* files as follow: Content of **/etc/skel/.bashrc**: {{:dev:environment:bashrc.txt|}} Create the following files in **/etc/skel/.bashrc.d/**:\\ bash_completion: {{:dev:environment:bash_completion.txt|}}\\ bashmarks: {{:dev:environment:bashmarks.txt|}}\\ defaults: {{:dev:environment:defaults.txt|}}\\ git: {{:dev:environment:git.txt|}}\\ penv: {{:dev:environment:penv.txt|}}\\ user-path: {{:dev:environment:user-path.txt|}} ++++ ---- ==== Create web developer user ==== > sudo useradd -m -g users -G www-data,sudo -s /bin/bash webdev > sudo passwd webdev Enter new UNIX password: Retype new UNIX password: === Setup the python development directory === Refer to [[http://serverfault.com/questions/6895/whats-the-best-way-of-handling-permissions-for-apache2s-user-www-data-in-var|this answer on serverfault]] to see how to setup permissions on the **/var/python** directory that will be used to store our scripts. ----- ===== Using a Shared Folder to Host your Projects ===== ----- === Creating a shared folder for your development environment === ----- You have to create a **shared folder** for your VM, then "prepare" it for correct symlinks operations from your **host** computer command line: See [[vms:webdev:apache#create_the_shared_www_folders|Create VirtualBox shared folder]] in this wiki. Test-mount your shared folder: > mount -t vboxsf Check your created (webdev) user uid and gid with: > id webdev uid=1001(webdev) gid=100(users) groups=100(users),27(sudo),33(www-data) Then create your python project directory and make it the target for the shared folder that we'll auto-mount at boot time by write to **/etc/fstab** as follow: > sudo mkdir /var/python > sudo nano /etc/fstab # automount python-share as root:www-data python-share /var/python vboxsf defaults,uid=1001,gid=33,dmode=775,fmode=664,umask=002 0 0 > sudo mount -a If the system crashes at boot time, you probably need to have the ''vboxsf'' module to be loaded early, before the mounting of file systems. Simply add a line containing ''vboxsf'' to ''/etc/modules'' and reboot, this should fix the problem. Another solution is to set ''noauto'' in ''/etc/fstab'' and manually mount drives in ''/etc/rc.local'', but other services are launched beforehand... Note that by default, your VirtualBox shared-folder won't let you create symlinks inside it, this willl prevent you from deploying python virtual environments inside the mounted shared folder (at some point a "read only system" error is generated, so only your projects files can be shared). **USING THE /etc/fstab AUTOMOUNT CAUSES THE DEBIAN 8 STARTUP TO DROP INTO EMERGENCY MODE :-(** === VirtualBox Symlinks Inside Shared Folders === ----- There is an, [[https://www.virtualbox.org/ticket/10085|apparently intentional]], limitation in VirtualBox that **prevents creation of symlinks from the VM inside a shared folder**. This seems specifically complex to resolve using a Windows host, you are invited to read the mentioned ticket for more details. Although on Linux and OS X hosts, using the following command should prevent you from experimenting the dreadful **Read-only file system** error while creating symlinks inside a shared folder on your guest VM. > VBoxManage setextradata VBoxInternal2/SharedFoldersEnableSymlinksCreate/ 1 This command has to be issued **from your host OS**, replacing //// with the name of your VirtualBox virtual machine and //// with this machine's shared folder name. You can verify that the command has been taken into account examining the output of: > VBoxManage getextradata enumerate ... Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/, Value: 1 ... Not doing so will **not let you install your Python virtual environments inside the shared folder**. In case you cannot reach a solution (under Windows probably), it is always possible for you to **have your virtual environments stored in a directory inside your VM that is NOT a shared folder**. ----- ===== Python and Django ===== ----- We'll need to run python scripts from our web interface, there are a couple of solutions to do this, but since FreeNAS is using Django, let's go for this one also... we'll hopefully learn something about FreeNAS along the way. [[http://djangoproject.com/|Django]] is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. We'll use uWSGI as web server gateway interface, [[http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html|following this guide...]]\\ We will set up uWSGI so that it creates a Unix socket, and serves responses to the web server via the WSGI protocol. ----- ==== Python 3 ==== Python 2.7(.10) is currently the default Python version that comes "pre-installed" with Debian Jessie (8). In case you're willing to use Python 3, here is the summary of the commands to execute, you can refer to [[http://linuxconfig.org/how-to-change-from-default-to-alternative-python-version-on-debian-linux|the reference article]] for more details. First let's make sure what Python version is default on the system and what versions are available: > python --version Python 2.7.10 > ls -l /usr/bin/python* lrwxrwxrwx 1 root root 24 Aug 7 16:12 /usr/bin/python -> python2.7 lrwxrwxrwx 1 root root 9 Mar 16 23:37 /usr/bin/python2 -> python2.7 -rwxr-xr-x 1 root root 3810568 Jul 1 14:37 /usr/bin/python2.7 Python 3 is NOT available on this system, let's install it, then make it default (using update-alternatives): > update-alternatives --list python update-alternatives: error: no alternatives for python > update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 > update-alternatives --install /usr/bin/python python /usr/bin/python3.4 2 > update-alternatives --list python /usr/bin/python2.7 /usr/bin/python3.4 > python --version Python 3.4.3+ We now have Python 3(.4.3) as our system's default. ---- ==== Python virtual environment (virtualenv) ==== A quick intro about Python virtual environments [[http://sametmax.com/les-environnement-virtuels-python-virtualenv-et-virtualenvwrapper/|can be found here (in french)]]. === setuptools & pip === setuptools is a Python code distribution tool, allowing, amongst other, automatic external libs installation and download from the Internet. It's like gem install under Ruby or npm for NodeJS. Python has its installation command called easy_install, picking in the [[http://pypi.python.org/pypi|pypi.python.org]] web site, like Cpan for Perl or Pear channels for PHP. Apparently pip does the same, but better. **NOTE**: It seems that on most distribution, if you're running Python 2.7.9+ or Python 3.4+, pip should already be available. This is not the case in Debian. With Python 2: > sudo apt-get install python-pip python-dev build-essential When using python 3: > sudo apt-get install python3-pip python3-dev build-essential Install virtualenv: > pip install virtualenv OR > pip3 install virtualenv === Setup virtual environment === > mkdir -p /var/python/virtenvs > cd /var/python/virtenvs > virtualenv fengui Activate the Python virtual environment: > source /var/python/virtenvs/fengui/bin/activate (fengui)webdev@djanginx:> Notice the virtualenv name in parenthesis in front of the prompt. To exit the Python virtual environment: (fengui)> deactivate ----- ==== Django ==== ----- Install Django into your virtualenv, create a new project, and cd into the project: (fengui)> pip install Django (fengui)> cd /var/python (fengui)> django-admin.py startproject fengui (fengui)> cd fengui To start a new app, after creating the project with Django, enter your project's directory and use: (fengui)> cd /var/python/fengui (fengui)> python manage.py startapp ----- ===== Basic uWSGI installation and configuration ===== ----- Inside your virtual environment: (fengui)> pip install uwsgi *** uWSGI is ready, launch it with /var/python/virtenv/fengui/bin/uwsgi *** === Basic test === Create a file called test.py (fengui)> cd /var/python/fengui (fengui)> nano test.py # test.py def application(env, start_response): start_response('200 OK', [('Content-Type','text/html')]) #return [b"Hello World"] # python3 return ["Hello World"] # python2 === Run uWSGI === (fengui)> uwsgi --http :8000 --wsgi-file test.py Then go to your VM IP address with a web browser: http://172.20.20.4:8000\\ You should see the "Hello World" page! This means we now have the following stack of components working: **the web client <-> uWSGI <-> Python** ----- ==== Django test site ==== ----- Now we want uWSGI to do the same thing, but to run a Django site instead of the test.py module. If you haven’t already done so, make sure that your gui project actually works: > cd /var/python/fengui > python manage.py runserver 172.20.20.4:8000 Then browse http://172.20.20.4:8000, you should see a message like:\\ It worked!\\ Congratulations on your first Django-powered page. And if that works, run it using uWSGI: > uwsgi --http :8000 --module fengui.wsgi module gui.wsgi: load the specified wsgi module Point your browser at the server; if the site appears, it means uWSGI is able to serve your Django application from your virtualenv, and this stack operates correctly: **the web client <-> uWSGI <-> Django** Now normally we won’t have the browser speaking directly to uWSGI. That’s a job for the webserver, which will act as a go-between. ----- ===== nginx ===== ----- ==== Install nginx ==== ----- Install and start nginx: > sudo apt-get install nginx > sudo service nginx start Browse to http://172.20.20.4, you should see a message like:\\ Welcome to nginx! These components of the full stack are working together: **the web client <-> the web server** ----- ==== Configure nginx for your site ==== ----- You will need the uwsgi_params file, which is available in the nginx directory of the uWSGI distribution, or from [[https://github.com/nginx/nginx/blob/master/conf/uwsgi_params]] Copy it into your project directory. In a moment we will tell nginx to refer to it. > cd /var/python/fengui > wget https://raw.githubusercontent.com/nginx/nginx/master/conf/uwsgi_params The uwsgi_params file contains this: uwsgi_param QUERY_STRING $query_string; uwsgi_param REQUEST_METHOD $request_method; uwsgi_param CONTENT_TYPE $content_type; uwsgi_param CONTENT_LENGTH $content_length; uwsgi_param REQUEST_URI $request_uri; uwsgi_param PATH_INFO $document_uri; uwsgi_param DOCUMENT_ROOT $document_root; uwsgi_param SERVER_PROTOCOL $server_protocol; uwsgi_param HTTPS $https if_not_empty; uwsgi_param REMOTE_ADDR $remote_addr; uwsgi_param REMOTE_PORT $remote_port; uwsgi_param SERVER_PORT $server_port; uwsgi_param SERVER_NAME $server_name; Now create a file called **fengui_nginx.conf**, and put this in it: # fengui_nginx.conf # the upstream component nginx needs to connect to upstream django { # server unix:///path/to/your/mysite/mysite.sock; # for a file socket server 127.0.0.1:8000; # for a web port socket (we'll use this first) } # configuration of the server server { # the port your site will be served on listen 80; # the domain name it will serve for server_name 10.0.7.212; # substitute your machine's IP address or FQDN charset utf-8; # max upload size client_max_body_size 75M; # adjust to taste # Django media location /media { alias /var/python/fengui/media; # your Django project's media files - amend as r$ } location /static { alias /var/python/fengui/static; # your Django project's static files - amend as $ } # Finally, send all non-media requests to the Django server. location / { uwsgi_pass django; include /var/python/fengui/uwsgi_params; # the uwsgi_params file you install$ } } This conf file tells nginx to serve up media and static files from the filesystem, as well as handle requests that require Django’s intervention. For a large deployment it is considered good practice to let one server handle static/media files, and another handle Django applications, but for now, this will do just fine. Symlink to this file from /etc/nginx/sites-enabled so nginx can see it: > sudo ln -s /var/python/fengui/fengui_nginx.conf /etc/nginx/sites-enabled/ ----- ==== Deploy static files ==== ----- Before running nginx, you have to collect all Django static files in the static folder. First of all you have to edit **/var/python/freenas/gui/gui/settings.py** adding: > nano /var/python/fengui/fengui/settings.py ADD: STATIC_ROOT = os.path.join(BASE_DIR, "static/") and then run > python manage.py collectstatic If you get an error like:\\ **ImportError: No module named django.core.management**\\ Make sure your virtualenv is activated. You can always see it between parenthesis at the start of the prompt line. Or you can issue the following command: > which python /usr/bin/python In this case it's the system installed python that gets executed as your virtualenv is disactivated, go to your virtualenv root and type: > cd /var/python/fengui > source bin/activate (freenas)> which python /var/python/fengui/bin/python ----- ==== Basic nginx test ==== ----- To check that media files are being served correctly, add an image called media.png to the **/var/python/fengui/media** directory, then visit http://your.ip.here:8000/media/media.png - if this works, you’ll know at least that nginx is serving files correctly. It is worth not just restarting nginx, but actually stopping and then starting it again, which will inform you if there is a problem, and where it is. > mkdir /var/python/fengui/media > cd /var/python/fengui/media > wget http://wiki.strategicz.com/vhyper/lib/exe/fetch.php?media=wiki:logo.png > mv fetch.php?media=wiki:logo.png media.png > service nginx stop > service nginx start Point your browser to http://172.20.20.4:8000/media/media.png, you’ll know at least that nginx is serving files correctly. ----- ===== nginx, uWSGI and test.py ===== ----- Let’s get nginx to speak to the “hello world” test.py application. (fengui)> cd /var/python/fengui (fengui)> uwsgi --socket :8000 --wsgi-file test.py This is nearly the same as before, except this time one of the options is different: socket :8000: use protocol uwsgi, port 8000 nginx meanwhile has been configured to communicate with uWSGI on that port, and with the outside world on port 80. Visit: http://172.20.20.4:80/ to check. And this is our stack: **the web client <-> the web server <-> the socket <-> uWSGI <-> Python** Meanwhile, you can try to have a look at the uswgi output at http://172.20.20.4:8000 - but quite probably, it won’t work because your browser speaks http, not uWSGI, though you should see output from uWSGI in your terminal. If you get an error like:\\ **no python application found, check your startup logs for errors**\\ in the command line and:\\ **Internal Server Error**\\ Make sure you issued the uwsgi command from the root of your project. ----- ===== Using sockets instead of TCP port ===== ----- So far we have used a TCP port socket, because it’s simpler, but in fact it’s better to use Unix sockets than ports - there’s less overhead. Edit **/var/python/projects/fengui/fengui_nginx.conf**, changing it to match: server unix:///var/python/fengui/fengui.sock; # for a file socket # server 127.0.0.1:8001; # for a web port socket (we'll use this first) Then restart nginx. > service nginx restart Run uWSGI again: > uwsgi --socket fengui.sock --wsgi-file test.py This time the socket option tells uWSGI which file to use. Try http://example.com:8000/ in the browser. === If this doesn't work === Check your nginx logs at **/var/log/nginx/error.log** If you see something like: ...connect() to unix:///var/python/freenas/gui/fengui.sock failed (13: Permission denied) while connecting to upstream... Try (very permissive): > uwsgi --socket fegui.sock --wsgi-file test.py --chmod-socket=666 or (more sensible): > uwsgi --socket fengui.sock --wsgi-file test.py --chmod-socket=664 You may also have to add your user to nginx’s group (which is probably www-data), or vice-versa, so that nginx can read and write to your socket properly. ----- ===== Running the Django application with uswgi and nginx ===== ----- Let’s run our Django application through a file socket using uwsgi: (virt)> uwsgi --socket fengui.sock --module fengui.wsgi --chmod-socket=664 OR (virt)> uwsgi --socket :8001 --module fengui.wsgi Now uWSGI and nginx should be serving up not just a “Hello World” module, but your Django project. The working stack is now: **the web client <-> the web server <-> the socket <-> uWSGI <-> Django** ----- ===== Tricks & Tips ===== ---- ==== Execution timeout ==== In case you have some scripts that require a **long** time to execute, you might want to change the //uwsgi_read_timeout// value in your nginx configuration file, in our above example this would be: > nano /var/python/fengui/fengui_nginx.conf ... location / { uwsgi_pass django_pub; include /var/python/fengui/uwsgi_params; uwsgi_read_timeout 180s; } ... You can read more about the **various nginx directives ruling the uwsgi module** on [[http://nginx.org/en/docs/http/ngx_http_uwsgi_module.html|this nginx website's page]]. Also you could have rules applied on specific locations: ... location /volumes { uwsgi_pass django_pub; include /var/python/fengui/uwsgi_params; uwsgi_read_timeout 180s; } ... ---- ===== There is more to learn ==== ----- === More on this subject === Consult the original article [[http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html#configuring-uwsgi-to-run-with-a-ini-file]] === Get going and learn django === [[https://docs.djangoproject.com/en/1.6/intro/tutorial01/]]