SSL setup scripts & documentation (#67)

* Testing an nginx/SSL reverse proxy docker image

* Custom nginx and certbot containers

* Change certbot volume mounts

* Figuring some things out

* Challenges have to be run over HTTP?

* Docker networking

* remove a volume

* Added nginx & certbot to docker compose - working on droplet

* Use native nginx templating

* SSH please

* SSH setup scripts

* Updated .env and documentation for setting up HTTPS

.env.template

@@ -8,3 +8,5 @@ POSTGRES_USERNAME=postgres
 POSTGRES_PASSWORD=postgrespw
 POSTGRES_SSLMODE=require
 CA_CERT=/usr/local/share/ca-certificates/ca-certificate.crt
+DOMAIN=oss.ebookfoundation.org
+SSL_EMAIL=example@gmail.com

README.md

@@ -7,23 +7,23 @@ The OAPEN Suggestion Service uses natural-language processing to suggest books b
 ## Table of Contents
 
 - [Installation (Server)](#installation-server)
-  * [DigitalOcean Droplet](#digitalocean-droplet)
+  - [DigitalOcean Droplet](#digitalocean-droplet)
-  * [DigitalOcean Managed Database](#digitalocean-managed-database)
+  - [DigitalOcean Managed Database](#digitalocean-managed-database)
-  * [Setup Users & Install Requirements](#setup-users-install-requirements)
+  - [Setup Users & Install Requirements](#setup-users-install-requirements)
-  * [Clone & Configure the Project](#clone-configure-the-project)
+  - [Clone & Configure the Project](#clone-configure-the-project)
-  * [SSL Certificate](#ssl-certificate)
+  - [SSL Certificate](#ssl-certificate)
 - [Running](#running)
 - [Logging](#logging)
 - [Endpoints](#endpoints)
-  * [/api](#get-api)
+  - [/api](#get-api)
-  * [/api/ngrams](#get-apingrams)
+  - [/api/ngrams](#get-apingrams)
-  * [/api/{handle}](#get-apihandle)
+  - [/api/{handle}](#get-apihandle)
-  * [/api/{handle}/ngrams](#get-apihandlengrams)
+  - [/api/{handle}/ngrams](#get-apihandlengrams)
 - [Service Components](#service-components)
-  * [Suggestion Engine](#suggestion-engine)
+  - [Suggestion Engine](#suggestion-engine)
-  * [API](#api)
+  - [API](#api)
-  * [Embed Script](#embed-script)
+  - [Embed Script](#embed-script)
-  * [Web Demo](#web-demo)
+  - [Web Demo](#web-demo)
 - [Updates](#updates)
 - [Local Installation (No Server)](#local-installation-no-server)
 
@@ -134,16 +134,67 @@ The OAPEN Suggestion Service uses natural-language processing to suggest books b
    POSTGRES_USERNAME=<Username of the postgres user>
    POSTGRES_PASSWORD=<Password of the postgres user>
    POSTGRES_SSLMODE=<'require' when using a managed database>
+   CA_CERT=<Path to the PostgreSQL ca-certificate.crt file>
+   DOMAIN=<domain the API should run on>
+   SSL_EMAIL=<email to send Certbot notifications to>
    ```
 
    > Postgres credentials can be found in the "Connection details" section of the managed database
 
 ### SSL Certificate
 
-   > Add information on how to retrieve certificate from DigitalOcean managed DB.
+#### SSL for Database
+
+> Add information on how to retrieve certificate from DigitalOcean managed DB.
 
 Create a directory in `api` called `certificates`. Once you have acquired a certificate for your managed database, copy it into `/api/certificates`. **Make sure that this file is named `ca-certificate.crt`, or ensure that the name of your certificate matches the `CA_CERT` variable in your `.env`.**
 
+#### SSL for API
+
+To setup SSL for the API endpoint, you need to first ensure you have the proper ports open, both in DigitalOcean's built-in firewall, and on the droplet itself using `ufw`. DigitalOcean's firewall is sufficient, so if you like you can just `sudo ufw disable`.
+
+If you'd like to keep both `ufw` and the DigitalOcean firewall running, enable the rules in `ufw`:
+
+```bash
+sudo ufw allow http
+sudo ufw allow https
+```
+
+Next, enable ports `80` and `443` in the DigitalOcean dashboard for the droplet. `443` is for HTTPS traffic and `80` is for HTTP traffic, which is needed for certbot to re-issue certificates when they expire. Don't worry, nginx will redirect all non-certbot traffic to HTTPS automatically.
+
+For certbot to issue an SSL certificate, your `DOMAIN` specified in `.env` must already have the proper DNS records pointing to the droplet's IPv4 address.
+
+Then, just make sure the scripts are executeable:
+
+```bash
+chmod +x setup-ssh.sh ready-ssh.sh
+```
+
+And run them in this order.
+
+```bash
+./setup-ssh.sh
+./ready-ssh.sh
+```
+
+> Wait for `setup-ssh.sh` to run to completion before running `ready-ssh.sh`.
+
+The API should now be accessible by HTTPS only at `https://<domain>/api`!
+
+However, to ensure that certificates are renewed before they expire, add a `cron` job that renews the certificate automatically. First, open the cron editor:
+
+```bash
+crontab -e
+```
+
+And add a line, replacing `/home/oapen/oapen-suggestion-service` with wherever you cloned the repository to locally:
+
+```
+0 5 1 */2 *    /usr/bin/docker compose up -f   /home/oapen/oapen-suggestion-service/docker-compose.yml certbot
+```
+
+Save your changes and exit. Now your certificates will renew automatically every 60 days!
+
 ## Running
 
 You can start the services by running the following command in the directory where you cloned the repo:
@@ -195,13 +246,10 @@ The array of books is ordered by the date they were added (most recent first).
 Any combination of the query parameters in any order are valid.
 
 - `/api?threshold=3`
-   
   Returns suggestions with a similarity score of 3 or more for the 25 most recently added books.
 - `/api?threshold=5&limit=100`
-   
   Returns suggestions with a similarity score of 3 or more for the 100 most recently added books.
 - `/api?limit=50&offset=1000`
-   
   Returns 50 books and all of their suggestions, skipping the 1000 most recent.
 
 ### GET /api/ngrams
@@ -220,13 +268,10 @@ The array of books is ordered by the date they were added (most recent first).
 Any combination of the query parameters in any order are valid.
 
 - `/api?limit=100`
-   
   Returns ngrams for the 100 most recent books.
 - `/api?offset=1000`
-   
   Returns ngrams for 25 books, skipping the 1000 most recent.
 
-
 ### GET /api/{handle}
 
 Returns suggestions for the book with the specified handle.
@@ -236,6 +281,7 @@ Returns suggestions for the book with the specified handle.
 `{handle}` (required): the handle of the book to retrieve.
 
 #### Query Parameters
+
 `threshold` (optional): sets the minimum similarity score to receive suggestions for. Default is 0, returning all suggestions.
 
 #### Examples
@@ -250,7 +296,6 @@ Returns suggestions for [the book](https://library.oapen.org/handle/20.500.12657
 
 Returns suggestions with a similarity score of 3 or more for [the book](https://library.oapen.org/handle/20.500.12657/37041) with the handle `20.400.12657/47581`.
 
-
 ### GET /api/{handle}/ngrams
 
 Returns the ngrams and their occurences for the book with the specified handle.

api/Dockerfile

@@ -17,6 +17,6 @@ COPY ./certificates/* /usr/local/share/ca-certificates/
 
 RUN chmod 644 /usr/local/share/ca-certificates/*.crt && update-ca-certificates
 
-EXPOSE 3001
+EXPOSE ${API_PORT}
 
 CMD [ "npm", "start" ]

api/app.js

@@ -1,4 +1,5 @@
 const express = require("express");
+const path = require("path");
 const app = express();
 
 const apiRoutes = require("./routes.js");
@@ -18,9 +19,9 @@ app.use("*", (req, res) => {
   return res.status(404).json({ error: "Resource not found" });
 });
 
-const port = process.env.API_PORT || 3001;
+const port = process.env.API_PORT || 8000;
 
 app.listen(port, () => {
   console.log("Suggestion Service API is up on port " + port);
-  console.log("Running at http://localhost:" + port + "/");
+  console.log("Running at http://localhost:" + port + "/api");
 });

api/db/data.js

@@ -7,7 +7,7 @@ async function querySuggestions(handle, threshold = 0) {
   await validate.checkHandle(handle);
 
   const query = new PQ({
-    text: `SELECT suggestion AS handle, score
+    text: `SELECT *
     FROM oapen_suggestions.suggestions
     WHERE handle = $1
     AND score >= $2`,

docker-compose.yml

@@ -10,12 +10,37 @@ services:
       - REFRESH_PERIOD=86400        # daily
       - HARVEST_PERIOD=604800       # weekly
   api:
+    container_name: api
     build: ./api/
     restart: always
     env_file:
       - .env
     ports:
-        - "0.0.0.0:${API_PORT}:${API_PORT}"
+      - 0.0.0.0:${API_PORT}:${API_PORT}
+    networks:
+      - nginx-passthrough
+  nginx:
+    image: nginx:mainline-alpine
+    restart: always
+    env_file:
+      - .env
+    volumes:
+      - ./nginx:/etc/nginx/templates
+      - /etc/certbot/conf:/etc/letsencrypt
+      - /etc/certbot/www:/var/www/certbot
+    ports:
+      - 80:80
+      - 443:443
+    networks:
+      - nginx-passthrough
+  certbot:
+    image: certbot/certbot
+    depends_on:
+      - nginx
+    volumes:
+      - /etc/certbot/conf:/etc/letsencrypt
+      - /etc/certbot/www:/var/www/certbot
+    command: certonly --webroot -w /var/www/certbot --force-renewal --email ${SSL_EMAIL} -d ${DOMAIN} --agree-tos
   web:
     build: ./web/
     restart: always
@@ -26,6 +51,6 @@ services:
     restart: always
     ports:
         - "0.0.0.0:${EMBED_SCRIPT_PORT}:3002"
-volumes:
+networks:
-  db:
+  nginx-passthrough:
-    driver: local
+    driver: bridge

nginx/nginx-challenge.conf

@@ -0,0 +1,12 @@
+server {
+    listen 80;
+
+    server_name ${DOMAIN} www.${DOMAIN};
+
+    location ~ /.well-known/acme-challenge {
+        allow all;
+        root /var/www/certbot;
+    }
+
+    return 301 https://${DOMAIN}$request_uri;
+}

nginx/nginx.conf

@@ -0,0 +1,23 @@
+server {
+    listen 80;
+
+    server_name ${DOMAIN} www.${DOMAIN};
+
+    location ~ /.well-known/acme-challenge {
+        allow all;
+        root /var/www/certbot;
+    }
+
+    return 301 https://${DOMAIN}$request_uri;
+}
+
+server {
+    listen 443 ssl http2;
+    ssl_certificate /etc/letsencrypt/live/${DOMAIN}/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/${DOMAIN}/privkey.pem;
+    server_name ${DOMAIN} www.${DOMAIN};
+
+    location / {
+        proxy_pass http://api:${API_PORT}/;
+    }
+}

nginx/nginx.conf.template

@@ -0,0 +1,23 @@
+server {
+    listen 80;
+
+    server_name ${DOMAIN} www.${DOMAIN};
+
+    location ~ /.well-known/acme-challenge {
+        allow all;
+        root /var/www/certbot;
+    }
+
+    return 301 https://${DOMAIN}$request_uri;
+}
+
+server {
+    listen 443 ssl http2;
+    ssl_certificate /etc/letsencrypt/live/${DOMAIN}/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/${DOMAIN}/privkey.pem;
+    server_name ${DOMAIN} www.${DOMAIN};
+
+    location / {
+        proxy_pass http://api:${API_PORT}/;
+    }
+}

ready-ssh.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+rm ./nginx/nginx.conf.template
+cp ./nginx/nginx.conf ./nginx/nginx.conf.template
+docker compose up --build -d nginx

setup-ssh.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+rm ./nginx/nginx.conf.template
+cp ./nginx/nginx-challenge.conf ./nginx/nginx.conf.template
+docker compose up --build -d nginx certbot