blog:2020:0830_django_uwsgi_setup [NervTech's Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Setting up a django project with uWSGI ======

{{tag>dev web django}}

In this article, we will see how to setup a django project with a uWSGI server for production usage. In the past I have only been using the internal development server provided directly in django for my test projects, but that development server should definitely not be used in production: instead, setting up a wsgi connection behing a proper web server such as Apache or nginx seems to be the recommanded way to proceed for production setup.

Also, in the process, I'm also now using a MariaDB backend for the database instead of a simple sqlite3 database: again, this seems to be the recommended path in production, so we will see how to set this up too.

====== ======

===== Initial setup of the django project =====

  * In this session, we used the article [[https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html|Setting up Django and your web server with uWSGI and nginx]] as main reference, so you might also want to check this.

<note>Another very good source of information on the uWSGI software is [[https://uwsgi-docs.readthedocs.io/en/latest/|the official documentation]] </note>

  * First think I noticed, was that my pip installation for python3 was out dated [//of course...//], so to upgrade it: <code>sudo pip3 install --upgrade pip</code>

<note>If pip3 is not installed at all, we can install it with the command: **sudo apt-get install python3-pip**</note>

  * Then we need to setup a virtual python env for our django project, which will be called **nvserver** in my case (and thus, I use that name too for the virtual env here): <sxh bash; highlight: []>cd $venvs_root_dir
virtualenv -p /usr/bin/python3 nvserver</sxh> 

  * We then need to "enter" or "activate" that virtual env: <sxh bash; highlight: []>source nvserver/bin/activate</sxh>

  * Once the virtualenv is activated we have to install some pip packages in that env: <sxh bash; highlight: []>    pip install Django
    pip install mysqlclient
    pip install uwsgi</sxh> 
  
  * Start django project: <sxh bash; highlight: []>cd $projects_root_dir
django-admin.py startproject nvserver</sxh>

  * Test running the project: <sxh bash; highlight: []>cd nvserver
python manage.py runserver</sxh>

<note>"runserver" didn't work the first time because port 8000 was already being used [and I also got warning about pending migrations]</note>

  * To run the test server (or development server if you will) on a different port and all available interfaces we can use: <code>python manage.py runserver 0.0.0.0:8090</code>

=> After changing the port I could finally get the project to run when opening the correponding web page (ie. http://192.168.2.20:8090 in my case)

<note>On my linux distribution I currently have django 2.2, more information on this specific release is available from [[https://docs.djangoproject.com/en/2.2/|the documentation for version 2.2 page]]</note>

===== Setting up the mariaDB database =====

  * For the setup of the database I use [[https://www.digitalocean.com/community/tutorials/how-to-use-mysql-or-mariadb-with-your-django-application-on-ubuntu-14-04|this article]] as base reference.

  * First we need to create the database/user with the following code: <code>mysql -u root -p -h 192.168.1.20 -P3307
CREATE DATABASE nvserver CHARACTER SET UTF8;
CREATE USER nvserveruser@localhost IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON nvserver.* TO 'nvserveruser'@'localhost';
GRANT ALL PRIVILEGES ON nvserver.* TO 'nvserveruser'@'192.168.1.20' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EXIT</code>

  * Then to set up the django project to use the mariadb database, we have to edit our **nvserver/settings.py** file: <sxh python; highlight: []>DATABASES = {
    # 'default': {
    #     'ENGINE': 'django.db.backends.sqlite3',
    #     'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    # }
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'nvserver',
        'USER': 'nvserveruser',
        'PASSWORD': 'password',
        'HOST': '192.168.1.20',
        'PORT': '3307',
    }
}</sxh>

<note>To be able to use the mysql database, we must also install the mysqlclient python module with: **pip install mysqlclient** as already mentioned above. Yet one thing I didn't mention is that to be able to install that package we first need to install the system package **libmysqlclient-dev** with apt/apt-get</note>

=> Once the database access is configured we can apply the migrations with: <sxh python; highlight: []>python manage.py makemigrations
python manage.py migrate
</sxh>

All the initial migrations went just fine for me. Then we can create the superuser: <sxh python; highlight: []>python manage.py createsuperuser</sxh>

When the superuser is created we can start the server again and then nagivate to http://192.168.1.20:8090/admin to check that we can indeed login with the superuser credentials: **OK**


===== Serving the django project with uWSGI =====

To be honest the first test I actually did on my side was to only install uwsgi **globally** (ie. system wide) with the command: <sxh bash; highlight: []>sudo -H pip3 install uwsgi</sxh>

Then I tried to serve my django project with it directly : <code>uwsgi --http :8090 --wsgi-file nvserver/wsgi.py</code>

But this didn't work as expected... [and in fact this makes complete sense]: when trying to load the file, the uwsgi server will produce an error because the django module is not found in the **global python environment**. So this means we should really run the uwsgi server from the virtual python env (and that's why we installed the package accordingly as reported above)

=> When executing the same command from the python virtual env, the uwsgi server will start properly, and we can navigate to http://192.168.1.20:8090 to see the django project running as desired.

Next we need to configure the webserver: <code>server {
  listen 80;
  server_name   api.nervtech.org;
  
  rewrite       ^ https://$server_name$request_uri? permanent;
}

# cf. https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html
# the upstream component nginx needs to connect to
upstream django {
    # server unix:///path/to/your/mysite/mysite.sock; # for a file socket
    server 192.168.1.20:8181; # for a web port socket (we'll use this first)
}

server {
    listen 443 ssl;
    server_name api.nervtech.org;
    charset     utf-8;

    ssl_certificate           /etc/letsencrypt/live/nervtech.org/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/nervtech.org/privkey.pem;

    ssl_prefer_server_ciphers on;
    ssl_protocols         TLSv1.2;

    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4';

    client_max_body_size 20m;

    access_log            /var/log/nginx/api.access.log;
    error_log            /var/log/nginx/api.error.log;

    add_header Strict-Transport-Security max-age=15768000;

    # Django media
    location /media  {
        alias /mnt/web/nvserver/media;  # your Django project's media files - amend as required
    }

    location /static {
        alias /mnt/web/nvserver/static; # your Django project's static files - amend as required
    }

    # Finally, send all non-media requests to the Django server.
    location / {
        uwsgi_pass  django;
        include     /mnt/web/nvserver/uwsgi_params; # the uwsgi_params file you installed
    }
}</code>


<note>To use this config we need to copy the **uwsgi_params** files available at [[https://github.com/nginx/nginx/blob/master/conf/uwsgi_params|this location]] into our project public web folder</note>

<note>Minor detail: from this step I started to use the port **8181** instead of **8090** ;-) Just don't get confused on this point.</note>

===== Collecting static files =====

My project setup is a bit non-standard I guess, but: I have the django project inside a git repository that I checkout on my host computer while my nginx server is running in a container that doesn't have access to the location where I checked out the project. So the idea I have in mind is to serve the django dynamic stuff with uWSGI while copying all the static files into a location that can be accessed by the nginx container afterwards. And it turns out this is very easy to do: you just need to specify an "out of source" STATIC_ROOT folder in the **settings.py** file: <sxh python; highlight: []>
# STATIC_ROOT = os.path.join(BASE_DIR, "static/")
STATIC_ROOT = "/mnt/array1/web/nvserver/static/"
</sxh>

Then we can collect all the static files with: <code>python manage.py collectstatic --clear --noinput</code>

This will fill the folder /mnt/array1/web/nvserver/static/ with content, and that content is then available from the nginx container as /mnt/web/nvserver/static/: all good for us :-)!

===== Establishing connection with uWSGI =====

At first I couldn't connect nginx with the uWSGI server, and I eventually realized this was due to the fact I was still using an "http" setup instead of a proper "socket" setup, so the command to start the server should be updated here to: <code>uwsgi --socket 192.168.1.20:8181 --wsgi-file nvserver/wsgi.py</code>

<note>Specifying the address for the socket in the command above seems to be needed: otherwise the uwsgi server might not use the correct interface by default, and that would also prevent nginx from establishing a connection to it</note>

===== Using Unix sockets instead of ports =====

To use a unix socket file instead of a socket port, we update our uwsgi start command again to something like: <code>uwsgi --socket /mnt/array1/web/nvserver/uwsgi.sock --wsgi-file nvserver/wsgi.py --enable-threads</code>

Then we update the nginx site config file to use that socket file: <code>upstream django {
    server unix:///mnt/web/nvserver/uwsgi.sock; # for a file socket
    # server 192.168.1.20:8181; # for a web port socket (we'll use this first)
}</code>

<note>Again, in my case the nginx server is running in a docker container, while the uwsgi server is running on the host. Yet it is possible to make a unix socket available to the docker container by sharing the folder where that socket is into a volume! (cf. [[https://superuser.com/questions/1411402/how-to-expose-linux-socket-file-from-docker-container-mysql-mariadb-etc-to|this page]] [except this is the inverted setup, but I think this should work anyway])</note>

**Important note**: We must also ensure that nginx has the permissions to access the socket file, so for my part I also had to add the command line option **%%--chmod-socket=666%%** to the uwsgi start command [//This is not really recommended, but I don't think this could be a problem somehow anyway//].

===== Start uWSGI with a config file =====

We now create a config file to start our django project uwsgi server: <code># nvserver_uwsgi.ini file
[uwsgi]

# Django-related settings
# the base directory (full path)
chdir           = /mnt/array1/dev/projects/NervSeed/python/django/nvserver
# Django's wsgi file
module          = nvserver.wsgi
# the virtualenv (full path)
home            = /mnt/array1/dev/projects/NervSeed/python/envs/nvserver

# process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 10
# the socket (use the full path to be safe
socket          = /mnt/array1/web/nvserver/uwsgi.sock
# clear environment on exit
vacuum          = true

chmod-socket    = 666
uid             = www-data
gid             = www-data
enable-threads  = true
</code>

Then we can ensure that the project is still working properly using the command line: <code>uwsgi --ini nvserver_uwsgi.ini</code>

<note>In the case defined above, we are actually executing the uwsgi module from the global python environment [//as far as I understand at least//]. So I believe that from this point we might not need the uwsgi package in the virtual env anymore ? [But I could certainly be wrong on this point...]</note>

And finally we can run uwsgi in **emperor mode** in the global python environment with: <code>uwsgi --emperor /mnt/array1/app_data/uwsgi --uid www-data --gid www-data</code>

===== Start uWSGI on system boot =====

To ensure that uWSGI is started when the system boots I added the command to my custom function "nv_start_all_services" with the additional content: <sxh bash; highlight: []>  logDEBUG "Starting uWSGI emperor"
  /usr/local/bin/uwsgi --emperor /mnt/array1/app_data/uwsgi --uid www-data --gid www-data --daemonize /mnt/array1/app_data/nginx_server/logs/uwsgi_emperor.log</sxh>

And finally we should call that script function in /etc/rc.local: <sxh bash; highlight: []>su kenshin -c "bash /home/kenshin/scripts/start_all_services.sh"</sxh>

With the script content: <sxh bash; highlight: []>lfile="/mnt/array1/admin/logs/nv_start_all_services.log"

# source /home/kenshin/scripts/profile.sh 2>&1 >> $lfile
source /home/kenshin/scripts/profile.sh
nv_start_all_services 2>&1 >> $lfile</sxh>

=> With this setup in place the uWSGI server is correctly started when the system boots, and will monitor any change on the config files inside /mnt/array1/app_data/uwsgi.

<note>In fact, to force restarting the uWSGI server after changing some python file one can simply touch the config file with for instance: **touch /mnt/array1/app_data/uwsgi/nvserver_uwsgi.ini** and this will force the "emperor" to reload the corresponding uwsgi "vassal"</note>

===== Conclusion =====

And this is it for the setup of the uWSGI server for django, in the end, this was less complex that I thought it would be ;-)! Now I just hope I could help clarifying some points on how to do it here.

Next thing I need now is to provide support to register/login users with my django app, but this is another story, so let's just stop here for this work session.