blog:2020:0830_django_uwsgi_setup

# Setting up a django project with uWSGI

In this article, we will see how to setup a django project with a uWSGI server for production usage. In the past I have only been using the internal development server provided directly in django for my test projects, but that development server should definitely not be used in production: instead, setting up a wsgi connection behing a proper web server such as Apache or nginx seems to be the recommanded way to proceed for production setup.

Also, in the process, I'm also now using a MariaDB backend for the database instead of a simple sqlite3 database: again, this seems to be the recommended path in production, so we will see how to set this up too.

Another very good source of information on the uWSGI software is the official documentation
• First think I noticed, was that my pip installation for python3 was out dated [of course…], so to upgrade it:
sudo pip3 install --upgrade pip
If pip3 is not installed at all, we can install it with the command: sudo apt-get install python3-pip
• Then we need to setup a virtual python env for our django project, which will be called nvserver in my case (and thus, I use that name too for the virtual env here):
cd $venvs_root_dir virtualenv -p /usr/bin/python3 nvserver • We then need to “enter” or “activate” that virtual env: source nvserver/bin/activate • Once the virtualenv is activated we have to install some pip packages in that env:  pip install Django pip install mysqlclient pip install uwsgi • Start django project: cd$projects_root_dir
django-admin.py startproject nvserver
• Test running the project:
cd nvserver
python manage.py runserver
“runserver” didn't work the first time because port 8000 was already being used [and I also got warning about pending migrations]
• To run the test server (or development server if you will) on a different port and all available interfaces we can use:
python manage.py runserver 0.0.0.0:8090

⇒ After changing the port I could finally get the project to run when opening the correponding web page (ie. http://192.168.2.20:8090 in my case)

On my linux distribution I currently have django 2.2, more information on this specific release is available from the documentation for version 2.2 page
• For the setup of the database I use this article as base reference.
• First we need to create the database/user with the following code:
mysql -u root -p -h 192.168.1.20 -P3307
CREATE DATABASE nvserver CHARACTER SET UTF8;
CREATE USER nvserveruser@localhost IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON nvserver.* TO 'nvserveruser'@'localhost';
GRANT ALL PRIVILEGES ON nvserver.* TO 'nvserveruser'@'192.168.1.20' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EXIT
• Then to set up the django project to use the mariadb database, we have to edit our nvserver/settings.py file:
DATABASES = {
# 'default': {
#     'ENGINE': 'django.db.backends.sqlite3',
#     'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
# }
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'nvserver',
'USER': 'nvserveruser',
'HOST': '192.168.1.20',
'PORT': '3307',
}
}
To be able to use the mysql database, we must also install the mysqlclient python module with: pip install mysqlclient as already mentioned above. Yet one thing I didn't mention is that to be able to install that package we first need to install the system package libmysqlclient-dev with apt/apt-get

⇒ Once the database access is configured we can apply the migrations with:

python manage.py makemigrations
python manage.py migrate


All the initial migrations went just fine for me. Then we can create the superuser:

python manage.py createsuperuser

When the superuser is created we can start the server again and then nagivate to http://192.168.1.20:8090/admin to check that we can indeed login with the superuser credentials: OK

To be honest the first test I actually did on my side was to only install uwsgi globally (ie. system wide) with the command:

sudo -H pip3 install uwsgi

Then I tried to serve my django project with it directly :

uwsgi --http :8090 --wsgi-file nvserver/wsgi.py

But this didn't work as expected… [and in fact this makes complete sense]: when trying to load the file, the uwsgi server will produce an error because the django module is not found in the global python environment. So this means we should really run the uwsgi server from the virtual python env (and that's why we installed the package accordingly as reported above)

⇒ When executing the same command from the python virtual env, the uwsgi server will start properly, and we can navigate to http://192.168.1.20:8090 to see the django project running as desired.

Next we need to configure the webserver:

server {
listen 80;
server_name   api.nervtech.org;

rewrite       ^ https://$server_name$request_uri? permanent;
}

# the upstream component nginx needs to connect to
upstream django {
# server unix:///path/to/your/mysite/mysite.sock; # for a file socket
server 192.168.1.20:8181; # for a web port socket (we'll use this first)
}

server {
listen 443 ssl;
server_name api.nervtech.org;
charset     utf-8;

ssl_certificate           /etc/letsencrypt/live/nervtech.org/fullchain.pem;
ssl_certificate_key       /etc/letsencrypt/live/nervtech.org/privkey.pem;

ssl_prefer_server_ciphers on;
ssl_protocols         TLSv1.2;

ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4';

client_max_body_size 20m;

access_log            /var/log/nginx/api.access.log;
error_log            /var/log/nginx/api.error.log;

# Django media
location /media  {
alias /mnt/web/nvserver/media;  # your Django project's media files - amend as required
}

location /static {
alias /mnt/web/nvserver/static; # your Django project's static files - amend as required
}

# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass  django;
include     /mnt/web/nvserver/uwsgi_params; # the uwsgi_params file you installed
}
}
To use this config we need to copy the uwsgi_params files available at this location into our project public web folder
Minor detail: from this step I started to use the port 8181 instead of 8090 Just don't get confused on this point.

My project setup is a bit non-standard I guess, but: I have the django project inside a git repository that I checkout on my host computer while my nginx server is running in a container that doesn't have access to the location where I checked out the project. So the idea I have in mind is to serve the django dynamic stuff with uWSGI while copying all the static files into a location that can be accessed by the nginx container afterwards. And it turns out this is very easy to do: you just need to specify an “out of source” STATIC_ROOT folder in the settings.py file:

# STATIC_ROOT = os.path.join(BASE_DIR, "static/")
STATIC_ROOT = "/mnt/array1/web/nvserver/static/"


Then we can collect all the static files with:

python manage.py collectstatic --clear --noinput

This will fill the folder /mnt/array1/web/nvserver/static/ with content, and that content is then available from the nginx container as /mnt/web/nvserver/static/: all good for us !

At first I couldn't connect nginx with the uWSGI server, and I eventually realized this was due to the fact I was still using an “http” setup instead of a proper “socket” setup, so the command to start the server should be updated here to:

uwsgi --socket 192.168.1.20:8181 --wsgi-file nvserver/wsgi.py
Specifying the address for the socket in the command above seems to be needed: otherwise the uwsgi server might not use the correct interface by default, and that would also prevent nginx from establishing a connection to it

To use a unix socket file instead of a socket port, we update our uwsgi start command again to something like:

uwsgi --socket /mnt/array1/web/nvserver/uwsgi.sock --wsgi-file nvserver/wsgi.py --enable-threads

Then we update the nginx site config file to use that socket file:

upstream django {
server unix:///mnt/web/nvserver/uwsgi.sock; # for a file socket
# server 192.168.1.20:8181; # for a web port socket (we'll use this first)
}
Again, in my case the nginx server is running in a docker container, while the uwsgi server is running on the host. Yet it is possible to make a unix socket available to the docker container by sharing the folder where that socket is into a volume! (cf. this page [except this is the inverted setup, but I think this should work anyway])

Important note: We must also ensure that nginx has the permissions to access the socket file, so for my part I also had to add the command line option --chmod-socket=666 to the uwsgi start command [This is not really recommended, but I don't think this could be a problem somehow anyway].

We now create a config file to start our django project uwsgi server:

# nvserver_uwsgi.ini file
[uwsgi]

# Django-related settings
# the base directory (full path)
chdir           = /mnt/array1/dev/projects/NervSeed/python/django/nvserver
# Django's wsgi file
module          = nvserver.wsgi
# the virtualenv (full path)
home            = /mnt/array1/dev/projects/NervSeed/python/envs/nvserver

# process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 10
# the socket (use the full path to be safe
socket          = /mnt/array1/web/nvserver/uwsgi.sock
# clear environment on exit
vacuum          = true

chmod-socket    = 666
uid             = www-data
gid             = www-data
enable-threads  = true

Then we can ensure that the project is still working properly using the command line:

uwsgi --ini nvserver_uwsgi.ini
In the case defined above, we are actually executing the uwsgi module from the global python environment [as far as I understand at least]. So I believe that from this point we might not need the uwsgi package in the virtual env anymore ? [But I could certainly be wrong on this point…]

And finally we can run uwsgi in emperor mode in the global python environment with:

uwsgi --emperor /mnt/array1/app_data/uwsgi --uid www-data --gid www-data

To ensure that uWSGI is started when the system boots I added the command to my custom function “nv_start_all_services” with the additional content:

  logDEBUG "Starting uWSGI emperor"
/usr/local/bin/uwsgi --emperor /mnt/array1/app_data/uwsgi --uid www-data --gid www-data --daemonize /mnt/array1/app_data/nginx_server/logs/uwsgi_emperor.log

And finally we should call that script function in /etc/rc.local:

su kenshin -c "bash /home/kenshin/scripts/start_all_services.sh"

With the script content:

lfile="/mnt/array1/admin/logs/nv_start_all_services.log"

# source /home/kenshin/scripts/profile.sh 2>&1 >> $lfile source /home/kenshin/scripts/profile.sh nv_start_all_services 2>&1 >>$lfile

⇒ With this setup in place the uWSGI server is correctly started when the system boots, and will monitor any change on the config files inside /mnt/array1/app_data/uwsgi.

In fact, to force restarting the uWSGI server after changing some python file one can simply touch the config file with for instance: touch /mnt/array1/app_data/uwsgi/nvserver_uwsgi.ini and this will force the “emperor” to reload the corresponding uwsgi “vassal”

And this is it for the setup of the uWSGI server for django, in the end, this was less complex that I thought it would be ! Now I just hope I could help clarifying some points on how to do it here.

Next thing I need now is to provide support to register/login users with my django app, but this is another story, so let's just stop here for this work session.

• blog/2020/0830_django_uwsgi_setup.txt