HTTPS Certificates

This page describes the HTTPS configuration for Savannah new servers (e.g. download0, frontend0, vcs0, mgt0). The current ('old') servers use a wildcard SSL certificate from GANDI.net. The new servers use Let's Encrypt certificates.

Further reading about Savannah server:

• SavannahArchitecture - overview of the current ('old') setup (i.e. vcs, mgt, frontend, internal, vcs, download).
• SavannahHosts - Configuration of the new servers (i.e. mgt0, vcs0, frontend0, internal0, download0).

Further reading about Savannah web UI:

• FrontEnd - Notes about the current frontend setup (git, cvs).
• FrontEndDevelopmentSite - Setting up development sites for Savannah.

EFF's CertBot

EFF's CertBot is a program that registers and renews "Let's Encrypt" certificates.

Savannah configuration uses the webroot method (aka certonly) in which the web server exposes a specific directory over HTTP. Certbot writes the authentication files to this directory and communicates with the "Let's Encrypt" certificate authority to register/renew certificates. For details, see https://certbot.eff.org/docs/using.html#webroot.

Savannah uses a custom certbot configuration which allows it to operate without requiring root access to the server. The client in default operation requires root superuser access in order to make modifications to the operating system configuration as an agent for the system administrator. The system modification is only needed during the initial installation. After that point it provides a vulnerable surface for attackers to aim at breaking through. By operating certbot without root privilege it removes that vulnerable security layer. Removing this vulnerable security attack service is worth the effort of doing the system administrator doing the manual configuration setup once. Details of this non-standard setup are described below.

Savannah non-root certbot installation

On Savannah we have installed a modified certbot recipe which allows certbot-auto to operate as a non-root user, isolated in its own directory. This setup is used on frontend0, vcs0, and mgt0.

Web Server Configuration, Part 1

In the Apache configuration file, expose this directory with a pre-set URL:

Alias /.well-known /home/certbot/www/certbot/.well-known
<Directory "/home/certbot/www/">
        AllowOverride None
        Require all granted
</Directory>

If using Nginx, use the following: (frontend0 does not yet use Nginx however mgt0 and vcs0 do.)

location /.well-known {
        root /home/certbot/www/ ;
}

Reload Apache or Nginx.

# service nginx reload
# service apache2 reload

Install fakeroot:

# apt-get install fakeroot

Create a dedicated non-root user:

# adduser --disabled-password --gecos Certbot certbot

Create required directories:

# su -l certbot
$ cd /home/certbot
$ mkdir -p bin etc www log tmp

Test the web setup by create a file and downloading it (the .well-known subdirectory is needed for testing, otherwise it will be automatically created by certbot on one the first run):

$ mkdir /home/certbot/www/.well-known
$ echo "this is a test file" > /home/certbot/www/.well-known/foo.txt
$ wget -O- http://frontend0.sv.gnu.org/.well-known/foo.txt
this is a test file
$ rm /home/certbot/certbot/.well-known/foo.txt  # clean up

Create two fake root programs (to simulate root environment for the certbot-auto program):

$ cd /home/certbot/bin

$ printf '#!/bin/sh\necho "fake apt-get" "$@"\nexit 0\n' > apt-get
$ chmod a+x apt-get

$ printf '#!/bin/sh\nfakeroot env "$@"' > sudo
$ chmod a+x sudo

$ wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

Verify that PATH is correctly set to process $HOME/bin prior to other directories. This ensures that the local copies will be called instead of the system ones and will override them. If this is not correct then fix the PATH setting in the $HOME/.profile file.

$ type apt-get
apt-get is /home/certbot/bin/apt-get

Run the certbot-auto command the first time using the --os-packages-only option. This will instruct it to call apt-get in order to install any dependencies that it requires. This will call our local fake apt-get which will simply print out the operation that it is attempting. Here is an example.

$ certbot-auto --no-self-upgrade --os-packages-only
Bootstrapping dependencies for Debian-based OSes...
fakeroot sudo apt-get update
fake apt-get update
fakeroot sudo apt-get install --no-install-recommends python python-dev virtualenv python-virtualenv gcc dialog libaugeas0 augeas-lenses libssl-dev libffi-dev ca-certificates
fake apt-get install --no-install-recommends python python-dev virtualenv python-virtualenv gcc dialog libaugeas0 augeas-lenses libssl-dev libffi-dev ca-certificates
Failed to install a working "virtualenv" command, exiting

The failure is expected since we are preventing it from being able to perform this root superuser operation. Note the apt-get command that it is trying to run. Perform that task as root yourself. In this case the example here is the following. The list of packages is system dependent and your list may very well be different from this example.

# apt-get install --no-install-recommends python python-dev ... use the list of packages from your output not this example

Run certbot-auto as certbot user with -d parameters to register new domains:

# su -l certbot
$ env XDG_DATA_HOME=src \
     certbot-auto --no-self-upgrade \
                  --email savannah-reports-private@gnu.org \
                  --agree-tos \
                  certonly --text --webroot \
                  --config-dir ~/etc \
                  --work-dir ~/tmp \
                  --logs-dir ~/log \
                  -w ~/www \
                  -d frontend0.savannah.gnu.org \
                  -d frontend0.savannah.nongnu.org \
                  -d frontend0.sv.gnu.org \
                  -d frontend0.sv.nongnu.org

Create the renew script.

$ printf '#!/bin/sh\nenv XDG_DATA_HOME=src certbot-auto --no-self-upgrade --email savannah-reports-private@gnu.org --agree-tos renew --text --webroot --config-dir ~/etc --work-dir ~/tmp --logs-dir ~/log -w ~/www\n' > $HOME/bin/renew-ssl-certificate
$ chmod a+x $HOME/bin/renew-ssl-certificate

Install the /etc/cron.daily/renew-ssl-certificate-cron script. FIXME: Provide a network location to download this small script. It is a little large for an inline copy here. However the main part of it is simply echo renew-ssl-certificate | su - certbot wrapped to log output and email root upon errors and significant changes.

The content of /home/certbot will look like this:

/home/certbot
|-- bin
|   |-- apt-get
|   |-- certbot-auto
|   |-- renew-ssl-certificate
|   `-- sudo
|
|-- etc [*** certificates and private keys in PEM format are stored here ***]
|   |-- archive
|   |-- keys
|   |-- live [*** certificates path ***]
|       [...]
|
|-- log [*** renewal logs will be saved here ***]
|
|-- src [*** the installed python virtualenv and 'certbot' program ***]
|   `-- letsencrypt
|       `-- bin
|       [...]
|
|-- tmp
|
`-- www
    `-- .well-known [*** domain validation directory exposed to web ***]

Web Server Configuration, Part 2

Forward Secrecy & Diffie Hellman Ephemeral Parameters

Both Apache and Nginx rely upon OpenSSL for the Diffie-Hellman which defaults to 1024 bits. Create a stronger 2048 parameter for the DH key exchange.

# cd /etc/ssl
# openssl dhparam -out dhparam.pem 2048

For Apache

<VirtualHost *:443>
        SSLEngine on
        SSLCertificateFile      /home/certbot/etc/live/frontend0.savannah.gnu.org/cert.pem
        SSLCertificateKeyFile   /home/certbot/etc/live/frontend0.savannah.gnu.org/privkey.pem
        SSLCertificateChainFile /home/certbot/etc/live/frontend0.savannah.gnu.org/chain.pem
        SSLOpenSSLConfCmd DHParameters "/etc/ssl/dhparams.pem"
        SSLHonorCipherOrder On
        SSLCipherSuite ... use current accepted values
        SSLProtocol ... use current accepted values
        include ssl_params.local;
        # Many more SSL-specific configuration items

For Nginx

server {
        listen 443 ssl;
        listen [::]:443 ssl;
        ssl_certificate /home/certbot/etc/live/frontend0.savannah.gnu.org/fullchain.pem;
        ssl_certificate_key /home/certbot/etc/live/frontend0.savannah.gnu.org/privkey.pem;
        ssl_dhparam /etc/ssl/dhparam.pem;
        ssl_prefer_server_ciphers On;
        ssl_ciphers ... use current accepted values
        ssl_protocols ... use current accepted values
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        include ssl_params.local;
        # Many more SSL-specific configuration

Frontend0 apache configuration

The majority of the Apache configurations are in /etc/apache2/sites-available/sv.inc. However this file does not contain SSL configuration - as it is included twice in other files (once for gnu, once for nongnu).

The file /etc/apache2/sites-available/vhosts-gnu.org contains:

<VirtualHost *:443>
  SSLEngine on

  SSLProtocol All -SSLv2 -SSLv3
  SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
  SSLHonorCipherOrder On

  SSLCertificateFile      /home/certbot/etc/live/frontend0.savannah.gnu.org/cert.pem
  SSLCertificateKeyFile   /home/certbot/etc/live/frontend0.savannah.gnu.org/privkey.pem
  SSLCertificateChainFile /home/certbot/etc/live/frontend0.savannah.gnu.org/chain.pem

  ServerName frontend0.savannah.gnu.org

  Include sites-available/sv.inc
</VirtualHost>

The file /etc/apache2/sites-available/vhosts-nongnu.org contains a configuration similar to the above, with a different ServerName statement (recall that the multiple domains gnu/nongnu,savannah/sv are registered in the same certificate file - so it is used in all these virtual hosts).

Registering additional sub-domains

Additional subdomains can be used for FrontEndDevelopmentSite.

The recommendation is to register new subdomains in a new, separate certificate file as opposed to adding more subdomains to the existing certficiate. This way development certificates are not exposed through the main production certificate information.

# su - certbot
$ env XDG_DATA_HOME=src \
    certbot-auto --no-self-upgrade --email savannah-reports-private@gnu.org \
                 --agree-tos certonly --text \
                 --webroot \
                 --config-dir ~/etc \
                 --work-dir ~/tmp \
                 --logs-dir ~/log \
                 -w ~/www \
                 -d jsmith.frontend0.savannah.gnu.org \
                 -d jsmith.frontend0.savannah.nongnu.org \
                 -d jsmith.frontend0.sv.gnu.org \
                 -d jsmith.frontend0.sv.nongnu.org

The certificate files for the new subdomain will be stored in /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org.

The appropriate apache configuration is:

<VirtualHost *:443>
  ServerName  jsmith.frontend0.savannah.gnu.org
  ServerAlias jsmith.frontend0.sv.gnu.org
  ServerAlias jsmith.frontend0.savannah.nongnu.org
  ServerAlias jsmith.frontend0.sv.nongnu.org

  [... other SSL related settings ...]
  SSLCertificateFile      /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org/cert.pem
  SSLCertificateKeyFile   /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org/privkey.pem
  SSLCertificateChainFile /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org/chain.pem
</VirtualHost>