====== Setting up a django project with uWSGI ======
{{tag>dev web django}}
In this article, we will see how to setup a django project with a uWSGI server for production usage. In the past I have only been using the internal development server provided directly in django for my test projects, but that development server should definitely not be used in production: instead, setting up a wsgi connection behing a proper web server such as Apache or nginx seems to be the recommanded way to proceed for production setup.
Also, in the process, I'm also now using a MariaDB backend for the database instead of a simple sqlite3 database: again, this seems to be the recommended path in production, so we will see how to set this up too.
====== ======
===== Initial setup of the django project =====
* In this session, we used the article [[https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html|Setting up Django and your web server with uWSGI and nginx]] as main reference, so you might also want to check this.
Another very good source of information on the uWSGI software is [[https://uwsgi-docs.readthedocs.io/en/latest/|the official documentation]]
* First think I noticed, was that my pip installation for python3 was out dated [//of course...//], so to upgrade it: sudo pip3 install --upgrade pip
If pip3 is not installed at all, we can install it with the command: **sudo apt-get install python3-pip**
* Then we need to setup a virtual python env for our django project, which will be called **nvserver** in my case (and thus, I use that name too for the virtual env here): cd $venvs_root_dir
virtualenv -p /usr/bin/python3 nvserver
* We then need to "enter" or "activate" that virtual env: source nvserver/bin/activate
* Once the virtualenv is activated we have to install some pip packages in that env: pip install Django
pip install mysqlclient
pip install uwsgi
* Start django project: cd $projects_root_dir
django-admin.py startproject nvserver
* Test running the project: cd nvserver
python manage.py runserver
"runserver" didn't work the first time because port 8000 was already being used [and I also got warning about pending migrations]
* To run the test server (or development server if you will) on a different port and all available interfaces we can use: python manage.py runserver 0.0.0.0:8090
=> After changing the port I could finally get the project to run when opening the correponding web page (ie. http://192.168.2.20:8090 in my case)
On my linux distribution I currently have django 2.2, more information on this specific release is available from [[https://docs.djangoproject.com/en/2.2/|the documentation for version 2.2 page]]
===== Setting up the mariaDB database =====
* For the setup of the database I use [[https://www.digitalocean.com/community/tutorials/how-to-use-mysql-or-mariadb-with-your-django-application-on-ubuntu-14-04|this article]] as base reference.
* First we need to create the database/user with the following code: mysql -u root -p -h 192.168.1.20 -P3307
CREATE DATABASE nvserver CHARACTER SET UTF8;
CREATE USER nvserveruser@localhost IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON nvserver.* TO 'nvserveruser'@'localhost';
GRANT ALL PRIVILEGES ON nvserver.* TO 'nvserveruser'@'192.168.1.20' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EXIT
* Then to set up the django project to use the mariadb database, we have to edit our **nvserver/settings.py** file: DATABASES = {
# 'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
# }
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'nvserver',
'USER': 'nvserveruser',
'PASSWORD': 'password',
'HOST': '192.168.1.20',
'PORT': '3307',
}
}
To be able to use the mysql database, we must also install the mysqlclient python module with: **pip install mysqlclient** as already mentioned above. Yet one thing I didn't mention is that to be able to install that package we first need to install the system package **libmysqlclient-dev** with apt/apt-get
=> Once the database access is configured we can apply the migrations with: python manage.py makemigrations
python manage.py migrate
All the initial migrations went just fine for me. Then we can create the superuser: python manage.py createsuperuser
When the superuser is created we can start the server again and then nagivate to http://192.168.1.20:8090/admin to check that we can indeed login with the superuser credentials: **OK**
===== Serving the django project with uWSGI =====
To be honest the first test I actually did on my side was to only install uwsgi **globally** (ie. system wide) with the command: sudo -H pip3 install uwsgi
Then I tried to serve my django project with it directly : uwsgi --http :8090 --wsgi-file nvserver/wsgi.py
But this didn't work as expected... [and in fact this makes complete sense]: when trying to load the file, the uwsgi server will produce an error because the django module is not found in the **global python environment**. So this means we should really run the uwsgi server from the virtual python env (and that's why we installed the package accordingly as reported above)
=> When executing the same command from the python virtual env, the uwsgi server will start properly, and we can navigate to http://192.168.1.20:8090 to see the django project running as desired.
Next we need to configure the webserver: server {
listen 80;
server_name api.nervtech.org;
rewrite ^ https://$server_name$request_uri? permanent;
}
# cf. https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html
# the upstream component nginx needs to connect to
upstream django {
# server unix:///path/to/your/mysite/mysite.sock; # for a file socket
server 192.168.1.20:8181; # for a web port socket (we'll use this first)
}
server {
listen 443 ssl;
server_name api.nervtech.org;
charset utf-8;
ssl_certificate /etc/letsencrypt/live/nervtech.org/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/nervtech.org/privkey.pem;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4';
client_max_body_size 20m;
access_log /var/log/nginx/api.access.log;
error_log /var/log/nginx/api.error.log;
add_header Strict-Transport-Security max-age=15768000;
# Django media
location /media {
alias /mnt/web/nvserver/media; # your Django project's media files - amend as required
}
location /static {
alias /mnt/web/nvserver/static; # your Django project's static files - amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
include /mnt/web/nvserver/uwsgi_params; # the uwsgi_params file you installed
}
}
To use this config we need to copy the **uwsgi_params** files available at [[https://github.com/nginx/nginx/blob/master/conf/uwsgi_params|this location]] into our project public web folder
Minor detail: from this step I started to use the port **8181** instead of **8090** ;-) Just don't get confused on this point.
===== Collecting static files =====
My project setup is a bit non-standard I guess, but: I have the django project inside a git repository that I checkout on my host computer while my nginx server is running in a container that doesn't have access to the location where I checked out the project. So the idea I have in mind is to serve the django dynamic stuff with uWSGI while copying all the static files into a location that can be accessed by the nginx container afterwards. And it turns out this is very easy to do: you just need to specify an "out of source" STATIC_ROOT folder in the **settings.py** file:
# STATIC_ROOT = os.path.join(BASE_DIR, "static/")
STATIC_ROOT = "/mnt/array1/web/nvserver/static/"
Then we can collect all the static files with: python manage.py collectstatic --clear --noinput
This will fill the folder /mnt/array1/web/nvserver/static/ with content, and that content is then available from the nginx container as /mnt/web/nvserver/static/: all good for us :-)!
===== Establishing connection with uWSGI =====
At first I couldn't connect nginx with the uWSGI server, and I eventually realized this was due to the fact I was still using an "http" setup instead of a proper "socket" setup, so the command to start the server should be updated here to: uwsgi --socket 192.168.1.20:8181 --wsgi-file nvserver/wsgi.py
Specifying the address for the socket in the command above seems to be needed: otherwise the uwsgi server might not use the correct interface by default, and that would also prevent nginx from establishing a connection to it
===== Using Unix sockets instead of ports =====
To use a unix socket file instead of a socket port, we update our uwsgi start command again to something like: uwsgi --socket /mnt/array1/web/nvserver/uwsgi.sock --wsgi-file nvserver/wsgi.py --enable-threads
Then we update the nginx site config file to use that socket file: upstream django {
server unix:///mnt/web/nvserver/uwsgi.sock; # for a file socket
# server 192.168.1.20:8181; # for a web port socket (we'll use this first)
}
Again, in my case the nginx server is running in a docker container, while the uwsgi server is running on the host. Yet it is possible to make a unix socket available to the docker container by sharing the folder where that socket is into a volume! (cf. [[https://superuser.com/questions/1411402/how-to-expose-linux-socket-file-from-docker-container-mysql-mariadb-etc-to|this page]] [except this is the inverted setup, but I think this should work anyway])
**Important note**: We must also ensure that nginx has the permissions to access the socket file, so for my part I also had to add the command line option **%%--chmod-socket=666%%** to the uwsgi start command [//This is not really recommended, but I don't think this could be a problem somehow anyway//].
===== Start uWSGI with a config file =====
We now create a config file to start our django project uwsgi server: # nvserver_uwsgi.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = /mnt/array1/dev/projects/NervSeed/python/django/nvserver
# Django's wsgi file
module = nvserver.wsgi
# the virtualenv (full path)
home = /mnt/array1/dev/projects/NervSeed/python/envs/nvserver
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
# the socket (use the full path to be safe
socket = /mnt/array1/web/nvserver/uwsgi.sock
# clear environment on exit
vacuum = true
chmod-socket = 666
uid = www-data
gid = www-data
enable-threads = true
Then we can ensure that the project is still working properly using the command line: uwsgi --ini nvserver_uwsgi.ini
In the case defined above, we are actually executing the uwsgi module from the global python environment [//as far as I understand at least//]. So I believe that from this point we might not need the uwsgi package in the virtual env anymore ? [But I could certainly be wrong on this point...]
And finally we can run uwsgi in **emperor mode** in the global python environment with: uwsgi --emperor /mnt/array1/app_data/uwsgi --uid www-data --gid www-data
===== Start uWSGI on system boot =====
To ensure that uWSGI is started when the system boots I added the command to my custom function "nv_start_all_services" with the additional content: logDEBUG "Starting uWSGI emperor"
/usr/local/bin/uwsgi --emperor /mnt/array1/app_data/uwsgi --uid www-data --gid www-data --daemonize /mnt/array1/app_data/nginx_server/logs/uwsgi_emperor.log
And finally we should call that script function in /etc/rc.local: su kenshin -c "bash /home/kenshin/scripts/start_all_services.sh"
With the script content: lfile="/mnt/array1/admin/logs/nv_start_all_services.log"
# source /home/kenshin/scripts/profile.sh 2>&1 >> $lfile
source /home/kenshin/scripts/profile.sh
nv_start_all_services 2>&1 >> $lfile
=> With this setup in place the uWSGI server is correctly started when the system boots, and will monitor any change on the config files inside /mnt/array1/app_data/uwsgi.
In fact, to force restarting the uWSGI server after changing some python file one can simply touch the config file with for instance: **touch /mnt/array1/app_data/uwsgi/nvserver_uwsgi.ini** and this will force the "emperor" to reload the corresponding uwsgi "vassal"
===== Conclusion =====
And this is it for the setup of the uWSGI server for django, in the end, this was less complex that I thought it would be ;-)! Now I just hope I could help clarifying some points on how to do it here.
Next thing I need now is to provide support to register/login users with my django app, but this is another story, so let's just stop here for this work session.