Saturday, 28 January 2017

// // 34 comments

Setting up Django Project Production Ready

Deploy Django Application with PostgreSQL, Nginx, Gunicorn, Virtualenv and Supervisor

Deploy Django Application project for Production Environment

Deploy Django Application on Linux Machine

Video Tutorial



Django is a powerful python web framework build for rapid development. It takes care of much of problems of web development, so you can only focus on writing your app rather than reinventing the wheel. The most interesting thing about Django is that it is free and open source.

So lets begin with the deployment procedure for any Django based application. The recommended configuration for running a Django application is a bit complex. It includes tools like: PostgreSQL, Nginx, Gunicorn, Virtualenv, Supervisord. This are advanced tools that makes the Django application more efficient.

There are some prerequisites for this tutorial, you must be ready with:

  1. A Debian based Linux machine with sudo privileges.
  2. A domain name server that will point to your server's IP.

Lets start the deployment process.

1. First start with the updating packages that are already available on server. So that server will be up to date. For this:

$ sudo apt-get update
$ sudo apt-get upgrade

2. Now, install postgreSQL and its required package.

$ sudo apt-get install postgresql postgresql-contrib

Lets create a database user and database for our Django application. For this first login as a postgres user, as by default only this user has privileges to create database in postgreSQL.

$ sudo su - postgres

Now, you are logged in as postgres user. lets create database user and assign necessary privileges to it.

postgres@ubuntu:~$ createuser --interactive -P
Enter name of role to add: db_user
Enter password for new role: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
postgres@ubuntu:~$

In above example, database user name is db_user. You can give the name as per your wish. Create database now. Give a appropraite name to database as well, as per your Django application.

postgres@ubuntu:~$ createdb --owner db_user django_db

here db_user is a databse user and django_db is a database name. Now logout from postgres user.

postgres@ubuntu:~$ logout
$

3. Till now our database setup is done. Now create a virtual environment for our Django application. For this we need to install, virtualenv package.

$ sudo apt-get install python-virtualenv

Now create a virtual environment.
Note: I am going to keep my django application in default folder that is /home/ubuntu/, So i will create my virtual environment here. If you want to keep your application in some other directory like /var/www/ then change your directory accordingly.

$ virtualenv -p python3 django_env

New python executable in django_env/bin/python
Installing distribute..............done.
Installing pip.....................done.

Here djano_env is the virtual environment name. Change it as per you, if required. To activate this environment, run

$ source django_env/bin/activate
(django_env) $

Now, install all the requirements of your project here, including django. Like:

(django_env) $ pip install django
Downloading/unpacking django
(...)
Installing collected packages: django
(...)
Successfully installed django
Cleaning up...

4. Now, clone your project directory, if you have already created project. Here I am creating a sample project for demo.

(django_env) $ django-admin.py startproject sample_project

Here project name is sample_project. Now check whether all your project requirement is setup or not by running it on development server. For this first navigate into your project and run the server.

(django_env) $ cd sample_project

(django_env) $ python manage.py runserver 0.0.0.0:8000
Validating models...
0 errors found
Django version 1.10.1, using settings 'hello.settings'
Jan 09, 2017 - 06:12:00
Development server is running at 0.0.0.0:8000/
Quit the server with CONTROL-C.

Our project is running on Server's public IP with port 8000. Now quit the server by pressing CONTROL-C.

5. Now, we need to configure postgreSQL so that it can communicate with our Django application. For this, install psycopg2 database adapter. But this adapter have some package dependencies, so first install them.

(django_env) $ sudo apt-get install libpq-dev python3-dev

Now install,

(django_env) $ pip install psycopg2

Now, configure database section in your project's settings.py file.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'django_db',
        'USER': 'db_user',
        'PASSWORD': '<password you entered when creating db_user>',
        'HOST': 'localhost',
        'PORT': '',                      # Set to empty string for default.
    }
}

Also add, your server's IP or domain name in settings.py as follow:

ALLOWED_HOSTS = ['<your server's IP or domain name>']

Without above setting, Django will not be accessible from your domain name or IP address. This is the security feature of Django.

Now, synchronise the database

(django_env) $ python manage.py migrate

Now our database setup is fully done.

6. Now we will setup our app server that is Gunicorn rather than using Django's by default single threaded development server.

(django_env) $ pip install gunicorn
Downloading/unpacking gunicorn
  Downloading gunicorn-0.17.4.tar.gz (372Kb): 372Kb downloaded
  Running setup.py egg_info for package gunicorn
Installing collected packages: gunicorn
  Running setup.py install for gunicorn
    Installing gunicorn_paster script to /webapps/hello_django/bin
    Installing gunicorn script to /webapps/hello_django/bin
Successfully installed gunicorn
    Installing gunicorn_django script to /webapps/hello_django/bin
Cleaning up...

Now you have gunicorn working, test it by running

(django_env) $ gunicorn sample_project.wsgi:application --bind 0.0.0.0:8001

Now, you can access gunicorn from your server's public IP with port 8001. Now to make gunicorn more useful for our django application, we need to configure its option. For this make a bash script called gunicorn.bash. You can change file name as per your choice.

(django_env) $ vim gunicorn_start.bash

now add following configurations into file

#!/bin/bash

NAME="django_app"                                   # Name of the application
DJANGODIR=/home/ubuntu/sample_project               # Django project directory
SOCKFILE=/home/ubuntu/django_env/run/gunicorn.sock  # we will communicte using this unix socket
USER=ubuntu                                         # the user to run as
GROUP=ubuntu                                        # the group to run as
NUM_WORKERS=3                                       # how many worker processes should Gunicorn spawn
DJANGO_SETTINGS_MODULE=sample_project.settings      # which settings file should Django use
DJANGO_WSGI_MODULE=sample_project.wsgi              # WSGI module name
echo "Starting $NAME as `whoami`"

# Activate the virtual environment

cd $DJANGODIR
source /home/ubuntu/django_env/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn't exist

RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your Django Unicorn
# Programs meant to be run under supervisor should not daemonize themselves (do not use --daemon)

exec gunicorn ${DJANGO_WSGI_MODULE}:application \
  --name $NAME \
  --workers $NUM_WORKERS \
  --user=$USER --group=$GROUP \
  --bind=unix:$SOCKFILE \
  --log-level=debug \
  --log-file=-

Here, all the variables are well explained in their comment part respectively. Change the variables value according to your configuration. Like NAME variable will define how you application will be identified in programs like ps, top etc. The recommended way to define no of workers is equals to 2*CPUs+1. For example, for a single CPU machine should be set with 3 workers.

Now make this script executable.

$ sudo chmod u+x gunicorn_start.bash

Lets test this script by running it.

(django_env) $ ./gunicorn_start.bash
Starting hello_app as hello
2013-06-09 14:21:45 [10724] [INFO] Starting gunicorn 18.0
2013-06-09 14:21:45 [10724] [DEBUG] Arbiter booted
2013-06-09 14:21:45 [10724] [INFO] Listening at: unix:/webapps/hello_django/run/gunicorn.sock (10724)
2013-06-09 14:21:45 [10724] [INFO] Using worker: sync
2013-06-09 14:21:45 [10735] [INFO] Booting worker with pid: 10735
2013-06-09 14:21:45 [10736] [INFO] Booting worker with pid: 10736
2013-06-09 14:21:45 [10737] [INFO] Booting worker with pid: 10737
^C (CONTROL-C to kill Gunicorn)

2013-06-09 14:21:48 [10736] [INFO] Worker exiting (pid: 10736)
2013-06-09 14:21:48 [10735] [INFO] Worker exiting (pid: 10735)
2013-06-09 14:21:48 [10724] [INFO] Handling signal: int
2013-06-09 14:21:48 [10737] [INFO] Worker exiting (pid: 10737)
2013-06-09 14:21:48 [10724] [INFO] Shutting down: Master
$ exit

7. Now, its time to setup supervisor so that it can supervise our application. If system reboots or application quits unexpectedly, supervisor will take care of its restart. For this, first install it.

$ sudo apt-get install supervisor

To supervise any program through supervisor, you need to create configuration file for that program inside /etc/supervisor/conf.d/ directory. For our Django application that is sample_project, we will create sample_project.conf

$ sudo vim /etc/supervisor/conf.d/sample_project.conf

Now, write following content into the opened file.

[program:sample_project]
command = /home/ubuntu/gunicorn_start.bash                  ; Command to start app
user = ubuntu                                                ; User to run as
stdout_logfile = /home/ubuntu/logs/gunicorn_supervisor.log   ; Where to write log messages
redirect_stderr = true                                       ; Save stderr in the same log
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8              ; Set UTF-8 as default encoding

Change the above configuration value according to your setup. As we mentioned in above file that the logs will be stored at /home/ubuntu/logs/gunicorn_supervisor.log, we need to make this directory and file.

(django_env) $ mkdir -p /home/ubuntu/logs/
(django_env) $ touch /home/ubuntu/logs/gunicorn_supervisor.log

After this done, we will ask supervisor to reread configuration files and update it so the our newly configuration file get add.

For Ubuntu 14.04:

$ sudo supervisorctl reread
sample_project: available

$ sudo supervisorctl update
sample_project: added process group

As you can see, our sample_project configuration file get added to supervisor process group. Now, start our app through it. For this

$ sudo supervisorctl start sample_project
sample_project: started

For Ubuntu 16.04:

$ sudo systemctl restart supervisor
$ sudo systemctl enable supervisor

To check status:

$ sudo supervisorctl status sample_project
sample_project                   RUNNING  pid 24768, uptime 0:00:10

To stop:

$ sudo supervisorctl stop sample_project
sample_project: stopped


To restart:

$ sudo supervisorctl restart sample_project
sample_project: stopped
sample_project: started

Now, our application will get automatically restart after system gets boot up or our application gets crashed.

8. Now, we need to setup last thing that is Nginx. So lets get start. Nginx will act as a server for our application. So first install it.

$ sudo apt-get install nginx

Now, we will need to create configuration file for our application inside /etc/nginx/sites-available/ directory. After this we will need to create symbolic link for it, in the /etc/nginx/sites-enabled directory. Lets do it one by one. First create configuration file.

$ sudo vim /etc/nginx/sites-available/sample_project.conf

Now, put following content into the opened file.

upstream sample_project_server {
  # fail_timeout=0 means we always retry an upstream even if it failed
  # to return a good HTTP response (in case the Unicorn master nukes a
  # single worker for timing out).
  server unix:/home/ubuntu/django_env/run/gunicorn.sock fail_timeout=0;
}

server {

    listen   80;
    server_name <your domain name>;

    client_max_body_size 4G;
    access_log /home/ubuntu/logs/nginx-access.log;
    error_log /home/ubuntu/logs/nginx-error.log;

    location /static/ {
        alias   /home/ubuntu/static/;
    }

    location /media/ {
        alias   /home/ubuntu/media/;
    }

    location / {

        # an HTTP header important enough to have its own Wikipedia entry:
        #   http://en.wikipedia.org/wiki/X-Forwarded-For
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


        # enable this if and only if you use HTTPS, this helps Rack
        # set the proper protocol for doing redirects:
        # proxy_set_header X-Forwarded-Proto https;

        # pass the Host: header from the client right along so redirects
        # can be set properly within the Rack application
        proxy_set_header Host $http_host;

        # we don't want nginx trying to do something clever with
        # redirects, we set the Host: header above already.
        proxy_redirect off;

        # set "proxy_buffering off" *only* for Rainbows! when doing
        # Comet/long-poll stuff.  It's also safe to set if you're
        # using only serving fast clients with Unicorn + nginx.
        # Otherwise you _want_ nginx to buffer responses to slow
        # clients, really.
        # proxy_buffering off;

        # Try to serve static files from nginx, no point in making an
        # *application* server like Unicorn/Rainbows! serve static files.
        if (!-f $request_filename) {
            proxy_pass http://sample_project_server;
            break;
        }
    }

    # Error pages
    error_page 500 502 503 504 /500.html;
    location = /500.html {
        root /home/ubuntu/static/;
    }
}

Change the configuration value according to your setup. Now create the symbolic link for it.

$ sudo ln -s /etc/nginx/sites-available/sample_project.conf /etc/nginx/sites-enabled/sample_project.conf

Now, start the Nginx.

$ sudo service nginx start

Now, if you browse your server's domain name into browser, you will get your site running. Congrats, your production ready django application is set. :)

34 comments:

  1. When i am running the command $ supervisorctl restart sample_project


    its giving me error http://localhost:9001 refused connection

    I am not getting whats wrong with this

    ReplyDelete
  2. Hello Saurabh,

    This error basically arises when the specified port is not open or being used by some other process. Try to check it by issuing following command:
    sudo netstat -nltp

    This command will show on which port, which process is running. If the port is free then try to check status of your app that whether it is running or not. You can check it with:
    sudo supervisorctl status sample_project

    If it is showing, "running" then try to check, log file which is stored in ~/logs/gunicorn_supervisor.log, it will tell you why it is happening.
    For more insights about the whole procedure, you can use video tutorial as well!
    Let us know, if your problem still persist after applying above solutions.

    ReplyDelete
  3. Gunicorn Permission Denied error while running the bash script.

    ReplyDelete
    Replies
    1. Hey, you probably have not given appropriate permission to file. Run following command mentioned in post:
      sudo chmod u+x gunicorn_start.bash

      Still, if you are getting same error then run:
      sudo chmod ugo+x gunicorn_start.bash

      This will solve your error. Let us know, if problem still persist.

      Delete
    2. still not working after applying above commands...my ubuntu user is mahesh and group is set to www-data..

      But if i remove user and group it works fine.

      Delete
    3. Try to run following command:
      sudo chown mahesh gunicorn_start.bash
      then:
      sudo chmod ugo+x gunicorn_start.bash

      It should solve your problem.

      Delete
  4. Application not running... showing this error in NGINX Logs.

    2017/04/04 13:07:25 [error] 2850#2850: *1 connect() failed (111: Connection refused) while connecting to upstream..

    How to solve this error.

    ReplyDelete
    Replies
    1. Complete Error in Log:

      2017/04/04 13:19:30 [crit] 3121#3121: *1 connect() to unix:/home/ubuntu/django/djangoenvironment/run/gunicorn.sock failed (2: No such file or directory) while connecting to upstream, client: 192.168.0.100, server: <your, request: "GET / / HTTP/1.1", upstream: "http://unix:/home/ubuntu/django/djangoenvironment/run/gunicorn.sock:/"

      Delete
    2. Can you please give me the output of "/home/ubuntu/logs/gunicorn_supervisor.log"?

      Delete
    3. Thanks the problem has been solved..I forgot to restart the supervisor after making some changes.

      Delete
    4. Nice! Thanks for using this post. Keep learning, keep sharing.

      Delete
  5. I have a problem with supervisor: 404

    (env) root@vz281558:~# systemctl start supervisor
    (env) root@vz281558:~# supervisorctl tail -f sample_project
    ==> Press Ctrl-C to exit <==
    unix:///var/run/supervisor.sock/logtail/sample_project/stdout Cannot read, status code 404


    My files:

    /etc/supervisor/conf.d/sample_project.conf

    [program:sample_project]
    command=/home/env/bin/gunicorn_start
    user=root
    stdout_logfile=/home/env/logs/gunicorn_supervisor.log
    redirect_stderr=true
    environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8


    /home/env/bin/gunicorn_start (I made this file as executable: chmod u+x gunicorn_start)

    #!/bin/bash

    NAME="myproject"

    DJANGODIR=/home/env/myproject/

    SOCKFILE=/home/env/myproject/run/gunicorn.sock

    USER=root
    GROUP=root
    NUM_WORKERS=3

    DJANGO_SETTINGS_MODULE=myproject.settings
    DJANGO_WSGI_MODULE=myproject.wsgi

    echo "Starting $NAME as `whoami`"

    cd $DJANGODIR
    source ../bin/activate
    export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
    export PYTHONPATH=$DJANGODIR:$PYTHONPATH

    RUNDIR=$(dirname $SOCKFILE)
    test -d $RUNDIR || mkdir -p $RUNDIR

    exec /home/env/bin/gunicorn ${DJANGO_WSGI_MODULE}:application --name $NAME --workers $NUM_WORKERS --user=$USER --group=$GROUP --bind=unix:$SOCKFILE --log-level=debug --log-file=-


    But when I run gunicorn_start directly: ./gunicorn_start I'm getting this:
    https://pastebin.com/embed_iframe/FUb83yMd

    Could you help me ?

    ReplyDelete
    Replies
    1. Its look like that your `gunicorn_start` script is running fine. I wonder why you are running command `supervisorctl tail -f sample_project`.
      Can you please provide the output of the file `/home/env/logs/gunicorn_supervisor.log`, It will help me look into your supervisor related issue.

      Also try to run:
      systemctl restart supervisor

      then:
      supervisorctl status sample_project

      And let me know the what it gives the output.

      Delete
    2. PROBLEM SOLVED !!!
      THANK YOU SO MUCH FOR THIS TUTORIAL !!! <3

      Delete
  6. Thanks for tut. But I was not able to load static files even after nginx config.

    ReplyDelete
    Replies
    1. Same problem here.....

      Delete
  7. work perfect nice step by step explanation :)

    ReplyDelete
  8. If I don't have server_name, what I have to put into /etc/nginx/sites-available/sample_project.conf??

    ReplyDelete
  9. Hello. Thank you for the tutorial. I'm having a problem with the gunicorn_start.bash file tho. It just prints Starting as jonas
    usage: gunicorn [OPTIONS] [APP_MODULE]
    gunicorn: error: argument -n/--name: expected one argument

    and does not work. Ḯ've tried to ask google, but nothing seems to help. Am i supposed to change the 'whoami' part?

    ReplyDelete
  10. Woah... Thank you very much... best simplified and easy tutorial to follow.. I got it all working after battling with tutorials from linode and digitalocean among many others without success all throughout yesterday .. You rock

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. I've got thos error in error.log of nginx : "2017/07/19 15:58:49 [emerg] 28550#0: host not found in upstream "www.mysite.com" in /etc/nginx/sites-enabled/mysite.conf:54" and this line refers to proxy_pass http://mysite.com; PLEASE HELP

    ReplyDelete
    Replies
    1. Hey, you are doing something wrong, as we are not referring www.mysite.com anywhere in the Nginx config. Just recheck your configuration file

      Delete
    2. is the proxy_pass correct ?

      Delete
    3. Yup. The Syntax is correct. But in tutorial, we are using socket file to connect with Application server that is Gunicorn.

      Delete
  13. I followed your tutoriel using my django application that i want to deploy, and i get an "400 Bad Request
    Request Header Or Cookie Too Large" any idea ? i tried many things but none worked

    ReplyDelete
    Replies
    1. Have you added, your IP address (or localhost address, if you are trying locally) in `ALLOWED_HOSTS` settings in Django settings file of your project as follow?
      ALLOWED_HOSTS= ['']

      Delete
    2. I did add my domain name as follow : ALLOWED_HOSTS= ['.name.com', 'www.name.com']

      Delete
    3. Replace .name.com with name.com and which domain you are using to access the Django application?

      Delete
  14. A great guide. I build my production server using your notes. Thx!
    One question. According to your instructions, after I tune gunicorn bash script and run it: gunicorn automaticaly creates socket with 777 permissions and under "ubuntu" user, which is owner of project folder. In same time usually nginx starts as www-data, so when nginx try to read from socket, he get permission from last 7 (All others). Many will say, such a wide permissions, not the best solution for production. Please tell me are there any opportunity to downgrade socket permisions from gunicorn (for example 660), so everything will stil work

    ReplyDelete
    Replies
    1. Hello Tk,
      By default, gunicorn creates socket file with 777 permission. We can only control user and group permission. You can use `--bind` option, so that socket will be available only from the bind values.
      Foe example:
      --bind unix:$SOCKFILE
      OR
      --bind 127.0.0.1:8000

      I hope, it will be useful for you.

      Delete
  15. I got through most of this tutorial and I know I'm very close. I tested gunicorn in the command line and it worked there. When I run '$ sudo supervisorctl start sample_project' I get an ERROR (spawn error). When I run 'sudo supervisorctl status sample_project' it returned FATAL Exited too quickly. The gunicorn_supervisor.log file contains this:
    EACCES
    supervisor: child process was not spawned.

    Any help is greatly appreciated!

    ReplyDelete
    Replies
    1. Does following command works fine?

      `./gunicorn_start.bash`

      Delete
    2. It was running into an error on line 27 saying command source not found. I had to modify the beginning of line 2 of sample_project.conf to 'command=bash ' to get the issue to disappear. I currently have nginx running but when I go to my domain i get a 404 not found when I expect to see my index page. Thanks for your help!

      Delete