I'm not sure it's possible to express how much I love Docker. My first Linux experience, with Redhat Manhattan, was set square in the middle of dependency hell. Dependency management has come a long way since then. However, this intensifies the effect of a dependency conflict when one does arise. Docker solves these problems and brings along both service and firewall management to boot. Because of this, I tend to develop and deploy with Docker.

For a web app to be useful, it necessarily needs to be accessible from the internet. Binding each app to the host negates much of the encapsulation Docker provides. So, we'll start by setting up a reverse-proxy into our Docker environment. In the end, our environment should resemble the following diagram.

Docker Compose

This setup will require docker, docker-compose, and cron on the host. Let's start by defining our Nginx container with docker-compose.

version: "3.8"

volumes:
  certs:
  config:
  sites:
  nginx:

networks:
  network:

services:
  nginx:
    image: nginx:latest
    networks:
     - network
    ports:
      - 443:8443
      - 80:8080
    restart: always
    tmpfs:
      - /run
      - /tmp
    volumes:
      - nginx:/etc/nginx
      - certs:/opt/certs
      - config:/opt/nginx
      - sites:/opt/sites

docker-compose.yml

As you can see, we've defined volumes for Nginx config and TLS certificates, a single network to attach to, and we're binding to ports 80 and 443 on the host. We've also mounted /tmp and /run in a memdisk. Pretty standard stuff.

And we're done, right? docker-compose up And, let's call it a day. 🙌

Not quite. This does run Nginx, but the out-of-the-box config isn't very useful. To make changes to the config we have to move files in and out of our container by hand which is error prone. Let's build some tooling around this so we don't have to remember all the fine details of this deployment every time we add or remove a virtual server, register a new TLS certificate, or whatever else we may need.

APP_ROOT = /opt/reverse-proxy
APP_NAME = reverse-proxy

etc/env

We'll start by creating a few directories and then define a few environment variables so we stay organized.

Nginx

Then let's start by initializing our docker volumes with the latest Nginx config. Then, mount all of our volumes in a temp container for initialization.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

sudo docker run --rm \
    --volume ${APP_NAME}_nginx:/etc/nginx/:rw \
    nginx:latest /bin/sh -c 'echo "Latest Nginx Config Copied"' \
;

echo "Entering configuration container" \
    && sudo docker run --rm \
        --volume ${APP_NAME}_nginx:/etc/nginx/:rw \
        --volume ${APP_NAME}_config:/opt/nginx:rw \
        --volume ${APP_NAME}_certs:/opt/certs:rw \
        --volume ${APP_NAME}_sites:/opt/sites:rw \
        --volume ${APP_ROOT}/etc/digitalocean.ini:/tmp/digitalocean.ini:ro \
        --volume ${APP_ROOT}/etc/init:/init:ro \
        archlinux:latest /init && \
echo "Exiting configuration container" \
;

bin/init

#!/bin/bash

REPO="https://github.com/harrisonapickettiv/nginx_config.git";

echo "Initializing reverse-proxy" \
    && echo "Installing git, this may take a while" \
        && pacman --noconfirm -Sy git > /dev/null \
    && echo "Installing Nginx config from ${REPO}" \
        && git clone ${REPO} /tmp/config \
        && cp /tmp/config/nginx.conf /etc/nginx/nginx.conf \
        && cp /tmp/config/{common_locations.conf,fastcgi.conf,tls.conf,cert.pem,key.pem} /opt/nginx \
        && openssl dhparam -out /opt/nginx/dhparams.pem 2048 \
    && echo "Installing certbot credentials" \
        && mkdir -p /opt/certs/secrets/ \
        && cp /tmp/digitalocean.ini /opt/certs/secrets/digitalocean.ini \
        && chmod 600 /opt/certs/secrets/digitalocean.ini \
;

etc/init

Our init script installs git in the temp container so we can pull an Nginx config repo (Maybe, I'll dissect my Nginx config in another post) and moves that config into place. Then, moves our certbot credentials into place and generates unique Diffie-hellman parameters for this deployment.

Now we've got a base Nginx config, but how do we add a server block or TLS certificate to our reverse-proxy? Let's write some scripts to handle these and other common tasks.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
NGINX_CONFIG=${1};
FILE=$(basename ${1});

sudo docker run --rm \
    --volume ${APP_NAME}_sites:/opt/sites \
    --volume ${NGINX_CONFIG}:/tmp/config/${FILE} \
    archlinux:latest \
        cp /tmp/config/${FILE} /opt/sites/ && echo "Copied config to Nginx" || exit 1 \
&& cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} \
    exec nginx /bin/sh -c 'nginx -s reload' \
&& echo "Nginx config reloaded successfully" || exit 1;

bin/site.add

server {
    server_name www.harrisonpickett.dev;
    access_log /dev/stdout;
    error_log stderr;

    include /opt/nginx/tls.conf;

    ssl_certificate /opt/certs/live/hpd/fullchain.pem;
    ssl_certificate_key /opt/certs/live/hpd/privkey.pem;
    ssl_trusted_certificate /opt/certs/live/hpd/fullchain.pem;

    client_max_body_size 0;

    location / {
        resolver 8.8.8.8 8.8.4.4 valid=30s;
        set $target https://hpd_blog:8080;
        proxy_pass $target;
    }

}

Nginx server block

To add a site to our reverse-proxy, we take the file path to an Nginx server block as a parameter. Then we launch a temp container mounting both the config fragment and our sites volume, copying the fragment into place. We then reload the Nginx config in the container with docker-compose. Pretty easy stuff, but stuff with fine details that could lead to errors if not automated.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
FILE=${1};

sudo docker run --rm \
    --volume ${APP_NAME}_sites:/opt/sites \
    archlinux:latest \
        rm -f /opt/sites/${FILE} && echo "Removed ${FILE} from Nginx config" || exit 1 \
&& cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} \
    exec nginx /bin/sh -c 'nginx -s reload' \
&& echo "Nginx config reloaded successfully" || exit 1;

bin/site.remove

To remove a site, we take the name of a server block as a parameter. We then launch a temp container with the sites volume mounted, remove the config fragment, and reload the Nginx config as before. Again easy, but the fine details could lead to problems if not automated.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

sudo docker run --rm \
    --volume ${APP_NAME}_sites:/opt/sites \
    archlinux:latest /bin/sh -c 'ls -1 /opt/sites' \
;

bin/site.list

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
FILE=${1};

sudo docker run --rm \
    --volume ${APP_NAME}_sites:/opt/sites \
    archlinux:latest cat /opt/sites/${FILE} \
;

bin/site.view

We should also provide tools to see what sites are installed and their configuration.

I think this looks pretty good. We have a repeatable setup and we can add sites to and remove sites from Nginx without having to remember all the fiddly details of our deployment config. Our app directory is starting to fill-out nicely.

Certbot

Now we'll configure EFF's certbot to generate TLS certificates with Let's Encrypt. I love Let's Encrypt. I used StartSSL for years and when they came to a fiery end, Let's Encrypt seemed to magically swoop in to save the day. 🤯

dns_digitalocean_token = 9563d7d3b95d4e8bb4682ec5d0f44be9bb53adb6e5b342f192b2459d3d246d24

etc/digitalocean.ini

Certbot has many options for automating certificate generation. We'll use the Digital Ocean DNS plugin to verify our identity. We'll place our API key in etc/digitalocean.ini. Scripts will need to be modified if another DNS provider is used.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
[ -n ${1} ] && CERT_NAME=${1};
[ -n ${1} ] && DOMAINS=${2};
[ -n ${2} ] && EMAIL=${3};

echo "Pulling new certbot image." \
    && sudo docker pull certbot/dns-digitalocean:latest > /dev/null;

sudo docker run --rm \
    --volume ${APP_NAME}_certs:/etc/letsencrypt \
    certbot/dns-digitalocean certonly \
        --agree-tos \
        --cert-name ${CERT_NAME} \
        --dns-digitalocean \
        --dns-digitalocean-credentials "/etc/letsencrypt/secrets/digitalocean.ini" \
        --domain ${DOMAINS} \
        --email ${EMAIL} \
        --non-interactive \
        --server https://acme-v02.api.letsencrypt.org/directory \
&& cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} \
    exec nginx /bin/sh -c 'nginx -s reload' \
&& echo "Nginx config reloaded successfully" || exit 1;

echo "Cleaning up docker images." \
    && sudo docker rmi certbot/dns-digitalocean:latest > /dev/null;

bin/cert.register

To generate a certificate, we take a certificate name, list of domains, and a contact email address as parameters. The certificate name is arbitrary, a label used by certbot to reference the certificate. The list of domains is a comma separated list of domains you wish to register. The first domain given becomes the certificate's Common Name, subsequent domains are listed in the Subject Alternative Name. Finally, the contact email address is used for expiry notification and account recovery. Certificates are placed on the certs volume, which is mounted to /opt/certs in the Nginx container.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
[ -n ${1} ] && CERT_NAME=${1};

echo "Pulling new certbot image." \
    && sudo docker pull certbot/dns-digitalocean:latest > /dev/null;

sudo docker run --rm \
    --volume ${APP_NAME}_certs:/etc/letsencrypt \
    certbot/dns-digitalocean delete \
        --cert-name ${CERT_NAME} \
        --non-interactive \
&& cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} \
    exec nginx /bin/sh -c 'nginx -s reload' \
&& echo "Nginx config reloaded successfully" || exit 1;

echo "Cleaning up docker images." \
    && sudo docker rmi certbot/dns-digitalocean:latest > /dev/null;

bin/cert.delete

To delete a certificate, we take the certificate name as a parameter. Certbot removes all the files related to the certificate. Caution: If any Nginx server blocks reference the certificate, Nginx will fail when the config is reloaded.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

echo "Pulling new certbot image." \
    && sudo docker pull certbot/dns-digitalocean:latest > /dev/null;

sudo docker run --rm \
    --volume ${APP_NAME}_certs:/etc/letsencrypt \
    certbot/dns-digitalocean renew \
        --dns-digitalocean \
        --dns-digitalocean-credentials "/etc/letsencrypt/secrets/digitalocean.ini" \
        --non-interactive \
        --server https://acme-v02.api.letsencrypt.org/directory \
&& cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} \
    exec nginx /bin/sh -c 'nginx -s reload' \
&& echo "Nginx config reloaded successfully" || exit 1;

echo "Cleaning up docker images." \
    && sudo docker rmi certbot/dns-digitalocean:latest > /dev/null;

bin/cert.renew

To renew all eligible certificates, simply run bin/cert.renew. Certbot will renew any installed certificates that expire in less than 30 days.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
[ -n ${1} ] && CERT_NAME=${1};

echo "Pulling new certbot image." \
    && sudo docker pull certbot/dns-digitalocean:latest > /dev/null;

sudo docker run --rm \
    --volume ${APP_NAME}_certs:/etc/letsencrypt \
    certbot/dns-digitalocean renew \
        --cert-name ${CERT_NAME} \
        --dns-digitalocean \
        --dns-digitalocean-credentials "/etc/letsencrypt/secrets/digitalocean.ini" \
        --force-renewal \
        --non-interactive \
        --server https://acme-v02.api.letsencrypt.org/directory \
&& cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} \
    exec nginx /bin/sh -c 'nginx -s reload' \
&& echo "Nginx config reloaded successfully" || exit 1;

echo "Cleaning up docker images." \
    && sudo docker rmi certbot/dns-digitalocean:latest > /dev/null;

bin/cert.renew.one

To force the renewal of a certificate, run bin/cert.renew.one passing the certificate name as a parameter. This will renew the specified certificate regardless of it's expiation date.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

sudo docker run --rm \
    --volume ${APP_NAME}_certs:/opt/certs \
    archlinux:latest ls \
        --ignore README \
        /opt/certs/live \
;

bin/cert.list

To list installed certificates, run bin/cert.list. This returns a list of certificate names.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;
[ -n ${1} ] && CERT_NAME=${1};

echo "Pulling new certbot image." \
    && sudo docker pull certbot/dns-digitalocean:latest > /dev/null;

sudo docker run --rm \
    --volume ${APP_NAME}_certs:/etc/letsencrypt \
    certbot/dns-digitalocean certificates \
        --cert-name ${CERT_NAME} \
;

echo "Cleaning up docker images." \
    && sudo docker rmi certbot/dns-digitalocean:latest > /dev/null;

bin/cert.view

To view detailed information about a certificate, run bin/cert.view passing the certificate name as a parameter. Certbot returns the certificate name, serial, key type, domains, and expiry date. The certificate and key path are returned as viewed from the Certbot container; swap /etc/letsencrypt/ with /opt/certs/ for use in an Nginx server block.

I think this is shaping up rather nicely. In addition to our repeatable config and site scripts, we can now generate and administer TLS certificates with Let's Encrypt.

Cron Daily

#!/bin/bash

docker pull certbot/dns-digitalocean:latest > /dev/null;

docker run --rm \
    --volume APP_NAME_certs:/etc/letsencrypt \
    certbot/dns-digitalocean renew \
        --dns-digitalocean \
        --dns-digitalocean-credentials "/etc/letsencrypt/secrets/digitalocean.ini" \
        --non-interactive \
        --server https://acme-v02.api.letsencrypt.org/directory \
&& cd APP_ROOT && \
docker-compose \
    --project-name APP_NAME \
    exec nginx /bin/sh -c 'nginx -s reload' \
|| exit 1;

docker rmi certbot/dns-digitalocean:latest > /dev/null;

etc/cert.cron

Let's automate our Certbot renewal script. To start, we'll strip all the sudo and echo bits from bin/cert.renew; these bits are for interactive use and not necessary when running from cron. Then we'll swap references to environment variables with tags we can replace with sed; we don't want our elevated script to run potentially unsecured code.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

sudo docker run --rm \
    --volume ${APP_NAME}_nginx:/etc/nginx/:rw \
    nginx:latest /bin/sh -c 'echo "Latest Nginx Config Copied"' \
;

echo "Entering configuration container" \
    && sudo docker run --rm \
        --volume ${APP_NAME}_nginx:/etc/nginx/:rw \
        --volume ${APP_NAME}_config:/opt/nginx:rw \
        --volume ${APP_NAME}_certs:/opt/certs:rw \
        --volume ${APP_NAME}_sites:/opt/sites:rw \
        --volume ${APP_ROOT}/etc/digitalocean.ini:/tmp/digitalocean.ini:ro \
        --volume ${APP_ROOT}/etc/init:/init:ro \
        archlinux:latest /init && \
echo "Exiting configuration container" \
;

echo "Installing certbot renew cron job" \
    && sed 's|APP_ROOT|'${APP_ROOT}'|g' ${APP_ROOT}/etc/cert.cron > /tmp/cert.cron \
    && sed -i 's|APP_NAME|'${APP_NAME}'|g' /tmp/cert.cron \
    && sudo sh -c '\
        chown root:root /tmp/cert.cron && \
        chmod 700 /tmp/cert.cron && \
        mv /tmp/cert.cron /etc/cron.daily/ \
    ' \
;

bin/init

Now, we'll add a block to the end of bin/init replacing our tags with the values set in etc/env during the install. Then we set permissions on the script and move it to /etc/cron.daily/. Now our renew script will check and renew certificates once a day.

Startup & Shutdown

There is an issue with Docker Compose that is nearly unforgivable. Docker Compose insists on starting in attached mode with no clean way to detach. To work around this, we'll script our start command and avoid using compose directly. We'll also script our stop command for the sake of consistency.

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} up \
    --detach \
;

bin/proxy.start

#!/bin/bash

. $(dirname $(realpath -s $0))/../etc/env;

cd ${APP_ROOT} && \
sudo docker-compose \
    --project-name ${APP_NAME} down \
;

bin/proxy.stop

This is the most basic of things, it's literally just calling docker-compose up -d. But considering the dire consequences if a mistake is made, it is best to modify our process and avoid docker-compose up completely.

Final Thoughts

And there we have it, a dockerized reverse-proxy server that we can hide our future projects and deployments behind.

We've scripted the most common actions required to make our proxy useful. This reduces the likelihood of mistakes and ensures consistency with our configuration over time.

If this project seems useful, check it out over on github. 😎