Savannah VM Hosts

A recent VM hosting migration caused IPv6 problems. Making a table to track the problem here.

| ping                                |
+-------------------------------------+
|Host1      |Host2      | IPv4 | IPv6 |
+-------------------------------------+
|download0  |internal1  | Okay | Okay |
|download1  |internal1  | Okay | Okay |
|internal1  |internal1  | Okay | Okay |
|mgt0       |internal1  | Okay | Okay |
|nfs1       |internal1  | Okay | Okay |
|vcs0       |internal1  | Okay | Okay |
|vcs1       |internal1  | Okay | Okay |
|vcs2       |internal1  | Okay | Okay |
+-------------------------------------+
|mgt0       |download0  | Okay | Okay |
|mgt0       |download1  | Okay | Okay |
|mgt0       |internal1  | Okay | Okay |
|mgt0       |nfs1       | Okay | Okay |
|mgt0       |vcs0       | Okay | Okay |
|mgt0       |vcs1       | Okay | Okay |
|mgt0       |vcs2       | Okay | Okay |
+-------------------------------------+
|download0  |nfs1       | Okay | Okay |
|download1  |nfs1       | Okay | Okay |
|internal1  |nfs1       | Okay | Okay |
|vcs0       |nfs1       | Okay | Okay |
|vcs1       |nfs1       | Okay | Okay |
|vcs2       |nfs1       | Okay | Okay |

| SSH port 22                         |
+-------------------------------------+
|Host1      |Host2      | IPv4 | IPv6 |
+-------------------------------------+
|mgt0       |download0  | Okay | Okay |
|mgt0       |download1  | Okay | Okay |
|mgt0       |internal1  | Okay | Okay |
|mgt0       |nfs1       | Okay | Okay |
|mgt0       |vcs0       | Okay | Okay |
|mgt0       |vcs1       | Okay | Okay |
|mgt0       |vcs2       | Okay | Okay |

| MySQL port 3306                     |
+-------------------------------------+
|Host1      |Host2      | IPv4 | IPv6 |
+-------------------------------------+
|download0  |internal1  | Okay | Okay |
|download1  |internal1  | Okay | FAIL |
|frontend1  |internal1  | Okay | Okay |
|frontend2  |internal1  | Okay | Okay |
|internal1  |internal1  | Okay | Okay |
|mgt0       |internal1  | Okay | Okay |
|nfs1       |internal1  | Okay | Okay |
|vcs0       |internal1  | Okay | Okay |
|vcs1       |internal1  | Okay | FAIL |
|vcs2       |internal1  | Okay | FAIL |
+-------------------------------------+
|download0  |internal1  | Okay | Okay |
|download1  |internal1  | Okay | FAIL |
|frontend1  |internal1  | Okay | Okay |
|frontend2  |internal1  | Okay | Okay |
|vcs0       |internal1  | Okay | Okay |
|vcs1       |internal1  | Okay | FAIL |
|vcs2       |internal1  | Okay | FAIL |

| HTTP port 80                        |
+-------------------------------------+
|Host1      |Host2      | IPv4 | IPv6 |
+-------------------------------------+
|mgt0       |download0  | Okay | Okay |
|mgt0       |vcs0       | Okay | Okay |
|mgt0       |vcs1       | Okay | Okay |
+-------------------------------------+
|external   |download   | Okay | Okay |
|external   |download0  | Okay | Okay |
|external   |git        | Okay | Okay |
|external   |vcs0       | Okay | Okay |
|external   |cvs        | Okay | Okay |
|external   |vcs1       | Okay | Okay |
|external   |savannah   | Okay | Okay |
|external   |frontend2  | Okay | Okay |

Current VM Hosts

The current hosts are:

download0
frontend1
internal1
mgt1
vcs0
vcs1
vcs2
nfs1

The frontend1 and frontend2 hosts alternate hosts that are intended to do the same thing as the other one. Production runs on one host while the other one is a testing candidate for the next upgrade. However they arrived at their present state in different ways causing some unintended differences. frontend1 was created as Trisquel 8 and then upgraded to Trisquel 9 and then upgraded to Trisquel 10. frontend2 was created as Triquel 9 and currently runs Trisquel 11.

These hosts are currently in process of being migrated to these new hosts:

download1

Further reading about the collection, perhaps more historical now:

SavannahArchitecture - overview of the current setup (i.e. vcs, mgt, frontend, internal, vcs).
SavannahServices - services/daemons available on savannah's hosts.
SavannahInternals - Savannah's inner-workings.

Git /etc etckeeper

In addition to these notes here all changes to /etc and therefore to the configuration are tracked by the 'etckeeper' package in git automatically as they happen. It checkpoints into version control the state of /etc before and after package installs and other changes. It checkpoints daily the state of all files. It is all automatic. No special actions need to be taken. You may however optionally make manual log entries but it is not required.

If you are trying to deduce what has been configured and these notes are not sufficient then checking the git log is also useful. However since the log comments are not useful you will need to look at what files have changed. These are examples of the type of browsing available.

cd /etc
git whatchanged
git whatchanged ./network
git whatchanged -p ./network

Host Common Elements

After pristine installation sevaral tasks are needed on any new host. Some things are common to all hosts. These common elements are documented here.

File /etc/apt/apt.conf.d/00local prevents Recommends from being automatically installed. These systems should be kept minimal and focused on the task at hand without extra peripheral packages.

Aptitude::Recommends-Important "false";
APT::Install-Recommends "0";
APT::Get::Show-Upgraded "true";
Acquire::PDiffs "false";

File /etc/apt/sources.list will contain the installation cdrom and dvd sources. This file should be cleaned and pointing to the close mirrors.

Update the host to the current packages.

apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove --purge && apt-get clean

Set up the etckeeper package. This package will track all changes to /etc in version control. It will also track what packages are installed and removed.

the Debian etckeeper package defaults to git and installs without error. Unfortunately the Trisquel etckeeper package defaults to bzr and fails to install without errors. But it can be worked around. Install git. Then install etckeeper. The installation of etckeeper on Trisquel will error and fail. After installation edit the etckeeper.conf file setting the version control system to git. Then manually finish the initialization.

apt-get install git
  ... setup /root/.gitconfig
apt-get install etckeeper
  ...setup /etc/etckeeper/etckeeper.conf
    VCS="git"
  etckeeper init
  etckeeper commit "Initial commit"
  etckeeper update-ignore

Install and configure needed locales.

dpkg-reconfigure locales

The VMs were handed to us including a /usr/sbin/policy-rc.d file that prevents services from starting during package upgrades. (It has no effect during boot time.) That was likely a mistake. That file is something that is most often used in chroots. This file must be removed.

rm -f /usr/sbin/policy-rc.d

Also the cloned ntpdate clock jumping script. Eventually need to set up a local tightly coupled peer pool. This following is mostly okay. The goal is to have at least three upstream servers and all of the VMs peering together.

rm -f /usr/local/bin/clock-sync.sh /etc/cron.d/ntpdate
apt-get install -y --purge ntp ntpdate-
echo peer 208.118.235.77 iburst >> /etc/ntp.conf
echo peer 208.118.235.78 iburst >> /etc/ntp.conf
echo peer 208.118.235.79 iburst >> /etc/ntp.conf
echo peer 208.118.235.200 iburst >> /etc/ntp.conf
echo peer 208.118.235.201 iburst >> /etc/ntp.conf
echo server 208.118.235.10 iburst >> /etc/ntp.conf
echo server PLACEHOLDER1 iburst >> /etc/ntp.conf
echo server PLACEHOLDER2 iburst >> /etc/ntp.conf
service ntp restart

Technically this fixes a bug in the locales package. On Trisquel it fails to create this file which is sourced by the default PAM configuration. On Debian this file is always created by the locales package. It shouldn't contain anything in our use of the file but it needs to exist to avoid errors from it failing to be sourced by PAM.

touch /etc/default/locale

Set logrotate to use date based names for archives instead of rolling depth numbers. Using date based names are better since then the file does not rolled to new names. That helps keep changes minimized. Backup doesn't need to backup the same file every time the name has changed. And so forth.

/etc/logrotate.d/dateext
  # Archive old versions of log files adding a daily extension like
  # YYYYMMDD instead of simply adding a number.
  dateext

Install generally useful utilities.

apt-get install gawk less screen htop wget dnsutils bind9-host
apt-get install heirloom-mailx

This includes Emacs. At the time of this writing version 24 is current and that number appears in the package names. At future times this version number will be the current version at that future time.

apt-get purge jove
apt-get install emacs emacs24-nox emacs24-el emacs24-common-non-dfsg emacs-goodies-el
apt-get install debian-el devscripts-el dpkg-dev-el

copy in a useful /root/.emacs

Set up /root/.profile appropriately.

mkdir /root/bin

/root/.profile # mgt has special copy with ChangeLog and other reminders
  LANG=en_US.UTF-8
  export LANG
  LC_COLLATE=C
  export LC_COLLATE
  PATH=$HOME/bin:$PATH

Set up ssh access:

/root/.ssh/authorized_keys (add mgt0, chmod u+w)

Bob finds this a required finger memory utility.

/usr/local/bin/ll
  #!/bin/sh
  exec ls -l "$@"

There are no hardwired terminals and therefore clearing the screen after exit provides no security but is harmful to use.

rm -f /etc/skel/.bash_logout

Various miscellaneous fixes.

...edit /etc/skel/.profile fix everything
...edit /etc/skel/.bashrc comment out --color
...comment out or remove all pre-defined aliases
...other fixes
...same for /root/.profile /root/.bashrc
.../root/.ssh/config

Set up /tmp as a tmpfs of moderate size.

/etc/fstab
  tmpfs  /tmp  tmpfs  size=1g  0  0

Check and fix if needed the /etc/hostname. Sometimes these are created with the FQDN and sometimes the short name. The current practice is to use the short name.

cat /etc/hostname
  ...possibly sed --in-place 's/\..*//' /etc/hostname

Configure the mailname.

echo mgt0.savannah.gnu.org > /etc/mailname

Fixup /etc/hosts so that 127.0.1.1 maps to the local hostname rather than the public IP address. Also it appears that the file is a leftover copy clone of another system. Clean out what isn't needed.

127.0.0.1       localhost

::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

127.0.1.1 mgt0.savannah.gnu.org mgt0

Install BIND caching nameserver. Ensure resolvconf installed.

apt-get install bind9 resolvconf
... install fixed /etc/resolvconf/update.d/resolvconf-update-bind
... fixed with a "forward only" statement
... /etc/default/bind9 set RESOLVCONF=yes
... set OPTIONS="-4 -u bind" since we are IPv4 only so far
ln -sfn /run/resolvconf/resolv.conf /etc/resolv.conf
  ... This may be best to let happen automatically after a reboot

Fixup /etc/network/interfaces for both IPv4 and IPv6.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 209.51.188.XX/24
        gateway 209.51.188.1
        dns-search savannah.gnu.org gnu.org

iface eth0 inet6 static
        pre-up echo 0 > /proc/sys/net/ipv6/conf/eth0/accept_dad
        address 2001:470:142:5::XX/48
        gateway 2001:470:142::1

The /etc/resolv.conf file should be managed by resolvconf but isn't apparently set correctly. Initially set this to bootstrap.

search savannah.gnu.org gnu.org
nameserver 208.118.235.95
nameserver 208.118.235.134
nameserver 46.43.37.71

Install postfix. Select "Internet site".

apt-get install --purge postfix

Bob's comment: I find Postfix to be more manageable than Exim. Therefore it is my choice to install Postfix. Others will choose Exim. The total capability of either is the same. In many ways it doesn't matter very much as long as the system has an MTA and that it is configured properly.

Normally no additional fixup is needed. But with the Trisquel 7 package there were errors during the initial installation which required corrective action. This appears to be Trisquel packaging bugs because they don't exist in the upstream Debian packages. It definitely is not a Postfix problem but exists in the packaging.

newaliases: warning: valid_hostname: misplaced delimiter: internal0.savannah.gnu.org.
newaliases: fatal: file /etc/postfix/main.cf: parameter myhostname: bad parameter value: internal0.savannah.gnu.org.
2016 Jul 26 17:25:55 internal0 fatal: file /etc/postfix/main.cf: parameter myhostname: bad parameter value: internal0.savannah.gnu.org.
dpkg: error processing package postfix (--configure): subprocess installed post-installation script returned error exit status 75

Fix this problem.

postconf -e myhostname=internal0.savannah.gnu.org
postconf -e 'mydestination=$myhostname, localhost.$mydomain, localhost'
postconf -X smtpd_relay_restrictions  # package cleanup
dpkg --configure -a

Those above problems needed to be fixed before any other configuration was done. Then additionally add other configuration.

postconf -e inet_interfaces=loopback-only
postconf -e masquerade_domains=savannah.gnu.org
postconf -e masquerade_exceptions=root
service postfix restart

The postfix inet_interfaces=loopback-only configuration is an extra level of security for leaf node systems that do not ever receive email. It configures to listen only on the loopback device making it completely isolated from the hostile Internet. It isn't required as the default configuration prevents relaying but is a good idea. Update: The latest image build included an alias for root to the FSF admins which should be removed first.

sed --in-place '/^root:/d' /etc/aliases
echo root: savannah-reports-private@gnu.org >> /etc/aliases
newaliases

Edit /etc/adduser.conf and comment out the EXTRA_GROUPS setting which for some reason the Trisquel package maintainer has uncommented. Otherwise every new user will be in a lot of extra groups that are normally only wanted when running on the console raw. Set the DIR_MODE umask back to a default sharing mode. All of the admins are on the same team working to admin these systems. They aren't general purpose for our own private use.

sed --in-place '/^EXTRA_GROUPS/s/^/#/' /etc/adduser.conf
sed --in-place '/^EXTRA_GROUPS/s/^/#/' /etc/adduser.conf
sed --in-place '/0751/s/0751/0755/' /etc/adduser.conf

Fix staff group on /usr/local which for some reason Trisquel/Ubuntu has removed. Put that back.

find /usr/local -exec chgrp staff {} +
find /usr/local -type d -exec chmod g+ws {} +

Ideally every host will have two IP addresses. One will be a stable RFC1918 private LAN address. The other will be a floating public IP address used for public services. This allows the IP address to float to a new node if needed. At the present time this does not exist but I am working on getting this available for our use.

Purge harmful utilities. The nscd breaks Unix ordering semantics and is too buggy to use. (The nscd ignores ordering of entries. The only way to have correct behavior is to avoid it. Trisquel seems to install it by default so it must be removed.)

apt-get purge nscd

Purge off the trisquel-base-data package which installs the file /etc/bash_completion.d/aliases which is too awful to allow to exist.

apt-get purge trisquel-base-data

Remove pkgsync as each machine has a unique list of packages, it isn't a cluster, and sync'ing them is not desired here.

apt-get purge pkgsync

Install more useful utilities.

apt-get install openssh-client openssh-server
ssh-keygen -t rsa

Since this will generate a new ssh key this new id_rsa.pub value must be configured elsewhere as needed.

Install the HAVEGE algorithm entropy collection daemon. This is only needed if no true hardware random number generator is connected. A True-RNG is much more desirable. They are now only $50. Try to obtain one if at all possible. Otherwise this is an acceptable fallback. Busy systems consume entropy.

apt-get install haveged

If this system is going to be monitored by the FSF Nagios system then their system is using this nagios module.

apt-get install xinetd check-mk-agent
  ...edit /etc/xinetd.d/check_mk, comment out: disable = yes

Using needrestart to audit processes that need restart.

apt-get install needrestart
  ...edit /etc/apt/apt.conf.d/99needrestart
  ...add -b option or it will spew to logged in consoles
  ...we run this tool interacively after upgrades

FIXME: Talk about vm.overcommit_memory=2. Talk about how much swap is needed. Talk about configuring the system for these.

FIXME: Set console / wall logging level.

FIXME: A bug is that SIGPIPE is ignored in cronjobs. Need to figure this out and fix it. It causes spurious "Broken pipe" errors. UPDATE: Only vcs0 and download0 are affected. mgt0, frontend0, and internal0 are okay and do not trap SIGPIPE. This means it probably isn't a Trisquel problem but is instead a customization made before the VMs were given to us.

Host mgt0

The IPv4 address for mgt0 is 208.118.235.77. It does not currently have an IPv6 address. This is okay since mgt0 is accessed only through fencepost. It currently does not have a floating address. This is also okay since the management node only very few public services.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 208.118.235.77/24
  gateway 208.118.235.1
  dns-search savannah.gnu.org gnu.org
  pre-up iptables-restore < /etc/default/iptables-rules || :

Install utilities useful on the management node.

apt-get install dsh

Use like this:

dsh -M -c -f hostlist somecommand someargs
dsh -M -c -f hostlist df -lh /

Set up Nginx for munin and for Let's Encrypt certbot and other uses.

apt-get install nginx

Set up munin-node.

apt-get install munin-node

Set up the munin master configuration.

apt-get install munin
...edit /etc/munin/munin.conf remove localhost entry
...edit /etc/munin/munin-conf.d/local.conf add host entries
[mgt0.savannah.gnu.org]
    address 127.0.0.1

[internal0.savannah.gnu.org]
    address 208.118.235.78

[frontend0.savannah.gnu.org]
    address 208.118.235.79

[vcs0.savannah.gnu.org]
    address 208.118.235.201

[download0.savannah.gnu.org]
    address 208.118.235.200

See https for details about SSL certificates setup using certbot/let's-encrypt on frontend0.

Set up DNS mastering of the Savannah DNS zones. This is described in its own DNS page.

Set up cronjobs for the routine Savannah maintenance tasks. (This eventually should be part of the Savannah scripted installation. For now the files need to be installed manually.) Note that at this instant the installation of sv_mailman has been hacked in place to call the /opt/savannah/sbin scripts instead of the previous hard coded paths to /usr/sbin/newlist and other Mailman utilities. This needs to be improved.

/etc/cron.d/savannah
/opt/savannah/bin/sv_mailman
/opt/savannah/sbin/change_pw
/opt/savannah/sbin/config_list
/opt/savannah/sbin/newlist
/opt/savannah/sbin/rmlist

To support the above the following Perl modules are needed.

apt-get install mysql-client libstring-random-perl libdate-calc-perl

Host internal0

The IPv4 address for internal0 is 209.118.188.78. It does not currently have an IPv6 address. This is okay since internal0 is accessed only through mgt0. It currently does not have a floating address. This is also okay since the management node only very few public services.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 208.118.235.77/24
  gateway 208.118.235.1
  dns-search savannah.gnu.org gnu.org
  pre-up iptables-restore < /etc/default/iptables-rules || :

Install MySQL server. In the future use MariaDB when it is available.

apt-get install mysql-server mysql-client

Change the port binding to be global rather than localhost so that the web frontend can connect to it remotely.

$EDITOR /etc/mysql/my.cnf
  ...bind-address 0.0.0.0

Set up the ~/.my.cnf file for access.

/root/.my.cnf
  [client]
  user            = root
  password        = ******
chmod go-rw /root/.my.cnf

Create the database. Add the local root at localhost user for command line and other access. Create the remote web frontend user and allow it to connect. Same user will connect from vcs to create new repositories. The nss-user and nss-root users are needed for the libnss-mysql-bg module used on vcs and download.

There is a choice here to use DNS names or to use hard coded IP addresses. DNS names are always simple and easier but that creates a dependency and a security dependency upon secure DNS. We do not yet have secure DNS implemented. This can be mitigated by ensuring that /etc/hosts contains every other host and IP address but that has other negatives. Or you can use IP addresses. Many will whine about that but it avoids a dependency and avoids the security concerns. Here I have chosen to use IP addresses. Migrations are infrequent and these are all centrally located and therefore easy to setup when needed.

FIXME: We went very quickly forward in the implementation and the list here is out of sync with what we actually did. Also we need to audit our list and remove some of the extra ones that we added in haste.

CREATE DATABASE savane CHARACTER SET utf8mb4;
GRANT ALL ON savane.* TO root@localhost;
GRANT ALL ON savane.* TO 'savannahscripts'@'$IPADDR' IDENTIFIED BY '********';
GRANT SELECT, SHOW VIEW ON savane.* TO 'nss-user'@'%' IDENTIFIED BY '********';
GRANT ALL ON savane.* TO 'nss-root'@'$VCSIP' IDENTIFIED BY '******';
GRANT ALL ON savane.* TO 'nss-root'@'$DOWNLOADIP' IDENTIFIED BY '******';
FLUSH PRIVILEGES;

The only write-access accounts are IP locked to the database server itself, the web UI frontend systems, and the administrative management system. One by local root socket access and the two remote clients by explicit IP address. The others are read-only access and to simplify configuration we are using a wildcard address. Note that network access to the database system is restricted by firewall rules so limiting them here is redundant and not needed.

Create a read-only user savane_ro to enable safer troubleshooting of database issues (see internal0:/root/agn/create-savane-readonly-user.sh for full details)

GRANT SELECT, SHOW VIEW ON savane. TO savane_ro@frontend0.savannah.gnu.org IDENTIFIED BY '*****';

To remove and discard an access revoke it.

REVOKE ALL ON savane.* FROM 'savannahscripts'@'209.51.188.188';
DROP USER 'savannahscripts'@'209.51.188.188';
FLUSH PRIVILEGES;

Set up munin-node. (libcache-cache-perl is needed for the extra mysql plugin.) Then install the extra mysql plugin with all of the suggested interfaces.

apt-get install munin-node libcache-cache-perl
cd /etc/munin/plugins
for i in $(/usr/share/munin/plugins/mysql_ suggest); do ln -s /usr/share/munin/plugins/mysql_ mysql_$i;done
echo 'allow ^208\.118\.235\.77$' >> /etc/munin/munin-node.conf
service munin-node restart

Install diffmon. Configure it for at least the following.

apt-get install diffmon

/root/ChangeLog : -u : savannah-hackers-private@gnu.org
/root/.ssh/authorized_keys : -u : savannah-hackers-private@gnu.org

Host frontend1

The IPv4 address for frontend1 is 209.51.188.72. It does not currently have an IPv6 address. This is needed to provide universal access to Savannah and is yet to be allocated. It also needs a private LAN address and a floating WAN IPv4 address also yet to be allocated.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 209.51.188.72/24
  gateway 209.51.188.1
  dns-search savannah.gnu.org gnu.org

Install MySQL client. In the future use MariaDB when it is available.

apt-get install mysql-client

Set up the ~/.my.cnf file for access.

/root/.my.cnf
  [client]
  host            = internal0
  user            = savannahscripts
  password        = ******
  database        = savane
chmod go-rw /root/.my.cnf

Install Apache and PHP7. As a future direction I (Bob Proulx) would much prefer this to be Nginx and FPM.

apt-get install apache2 apache2-mpm-prefork
apt-get install lynx  # for "apachectl status"
apt-get install php7.4 php7.4-cli php7.4-fpm php7.4-gd php7.4-mysql libapache2-mod-php7.4
apt-get install imagemagick gettext # used by php installation script
apt-get install passwdqc # used by php code to check password strength
apt-get install python python-pip python-dev # for future developments
apt-get install libmysqlclient-dev # for future developments

Since this is a migration I am going to take the easy documentation way out and say migrate the hosts from the old frontendX to the new frontendY without specifying them here.

/etc/apache2/sites-available/*.conf

Here is a grep of the VirtualHost sites that are better documented in their own configuration files than here.

/etc/apache2/sites-enabled/00-default-local.conf:<VirtualHost _default_:80>
/etc/apache2/sites-enabled/00-default-local.conf:<VirtualHost _default_:443>
/etc/apache2/sites-enabled/sv-i18n-dev.conf:<VirtualHost *:80>
/etc/apache2/sites-enabled/sv-i18n-dev.conf:  ServerName   i18n.savannah.gnu.org
/etc/apache2/sites-enabled/sv-i18n-dev.conf:<VirtualHost *:443>
/etc/apache2/sites-enabled/sv-i18n-dev.conf:  ServerName  i18n.savannah.gnu.org
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:<VirtualHost *:80>
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:  ServerName savannah.gnu.org
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:<VirtualHost *:80>
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:  ServerName sv.gnu.org
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:<VirtualHost *:443>
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:  ServerName savannah.gnu.org
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:<VirtualHost *:443>
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:  ServerName sv.gnu.org
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:#<VirtualHost *:443>
/etc/apache2/sites-enabled/vhosts-gnu.org.conf:#  ServerName sv.gnu.org
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:<VirtualHost *:80>
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:  ServerName savannah.nongnu.org
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:<VirtualHost *:80>
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:  ServerName sv.nongnu.org
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:<VirtualHost *:443>
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:  ServerName savannah.nongnu.org
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:<VirtualHost *:443>
/etc/apache2/sites-enabled/vhosts-nongnu.org.conf:  ServerName sv.nongnu.org

Set up the HTTP SSL certificates. We are using Let's Encrypt DV Domain Validation certificates. We are using the dehydrated client.

root@frontend1:~# find /etc/dehydrated/ /etc/cron.daily/renew-https-cert-local -type f -ls
   394555      4 -rw-r--r--   1 root     root          141 Feb 10  2020 /etc/dehydrated/config
   394561      4 -rw-r--r--   1 root     root          345 Feb 10  2020 /etc/dehydrated/conf.d/local.sh
   396695      4 -rw-r--r--   1 root     root          213 Jul  9  2020 /etc/dehydrated/domains.txt
   268207      8 -rwxr-xr-x   1 root     root         4710 May 26  2020 /etc/cron.daily/renew-https-cert-local

Set up munin-node.

apt-get install munin-node
echo 'allow ^209\.51\.188\.77$' >> /etc/munin/munin-node.conf
echo 'allow ^2001:470:142:5::77$' >> /etc/munin/munin-node.conf
service munin-node restart

Create directories for savannah's project-submissions and tracker attachments, ensure the webserver can write in them. Place these on the NFS mount for shared access.

root@frontend1:~# ls -dl /var/lib/savane
lrwxrwxrwx 1 root root 23 Dec  9  2019 /var/lib/savane -> /net/vcs/var.lib.savane

root@frontend1:~# ll -dl /var/www/submissions_uploads
lrwxrwxrwx 1 root root 43 Jan  7 19:17 /var/www/submissions_uploads -> /net/vcs/var.lib.savane/submissions_uploads

root@frontend1:~# grep nfs /etc/fstab
nfs1:/srv/vcs  /net/vcs nfs4  defaults,async,nofail

For reference they were created like this but in their previous locations.

for i in /var/www/submissions_uploads \ /var/lib/savane/trackers_attachments ; do \ mkdir -p $i ; \ chgrp www-data $i ; \ chmod g+w $i ; \ done

See FrontEndSetup, FrontEndDevelopmentSite for details about Apache/PHP/db configuration from the frontend's website.

See HowToAdminThisWiki for wiki-related configuration settings on frontend1.

We are currently using the package default values for the MaxRequestWorkers and MaxSpareThreads. The PHP processes for Savane are sufficiently light and usage not having been a problem that we haven't found the need to calculate the actual limits needed and have not set them.

Host vcs0

The vcs0 host NFS mounts the data from the NFS server. At the time of this writing the NFS server is the previous vcs host. The hope is that a new VM for data0 will be created to host the data. But there is no disk space for that VM. Therefore we are compromising.

mkdir /net /net/vcs
/etc/fstab contains
... vcs:/ /net/vcs nfs  defaults,async 0 0
echo 208.118.235.72 vcs.savannah.gnu.org vcs >> /etc/hosts
... above /etc/hosts entry works around boot ordering bug

After mounting check /proc/mounts and verify that rsize and wsize have been negotiated to be "large enough". Currently it is being negotiated as rsize=1048576,wsize=1048576.

Set up munin-node.

apt-get install munin-node
echo 'allow ^208\.118\.235\.77$' >> /etc/munin/munin-node.conf
service munin-node restart

Install required packages:

apt-get install imagemagick gettext # used by savane php installation script
apt-get install vim

At least one version control hook script doesn't have a way to avoid sending anemail that isn't desired to be sent. Therefore we send it to the nobody user which is the traditional Unix /dev/null user. However these days distros don't default all of those to root. Change nobody to /dev/null in order to provide this again.

sed 's/^nobody[ ]*:.*/nobody: /dev/null/' /etc/aliases

Install required perl modules. This module is not packaged. We should upgrade to the currently supported module but just want to keep the same for the moment. Using cpan like this as group staff (not as root) will install it in the /usr/local tree and make it available to the Perl on the system.

cpan -i Mail::Send   # used by perl backend script 'sv_groups'

See BackEndSetup for details about setting up the savannah's custom perl/python backend scripts.

Set up /etc/cron.d/savannah with crons to update the canonical email address map to map local user email addresses to their registered email address.

/opt/savannah/bin/sv_aliases --cron && cd /etc/postfix && make

(Notes: This requires $sys_cron_mail="yes"; in the savane.conf.pl file. It creates /etc/email-aliases, the old Exim map file, and rewrites /etc/aliases. Then it uses those to create /etc/postfix/canonical, the Postfix map file, to map out going email to the registered user email instead of the at savannah.gnu.org address.)

The Trisquel PAM configuration sets max retries at 3 while ssh by default uses 6. This causes a config where pam will start warning about excess attempts when ssh should be reacting to the status passed through pam. The fix seems to be a newer ssh but we are on the Trisquel LTS security stream. A workaround is to tell ssh to limit the number of retries.

File /etc/ssh/sshd_config
# Prevent endless of these messages being logged.
#   2017 Jan 14 21:52:48 vcs0 PAM service(sshd) ignoring max retries; 6 > 3
MaxAuthTries 3

Prevent ssh from passing through LANG and LC_* so as to avoid ungenerated locales on the local server from being seen by the script used for access control.

File /etc/ssh/sshd_config
#AcceptEnv LANG LC_*
...sshd defaults to not accepting any environment variables

Subsystem Rsync

All of the version control services use rsync to make raw copies of the repositories available for download. This is common to all.

apt-get install rsync xinetd

...edit /etc/xinetd.d/rsync
service rsync
{
        socket_type     = stream
        wait            = no
        user            = nobody
        server          = /usr/bin/rsync
        server_args     = --daemon
}

Subsystem Bazaar BZR

Rsync repository access

...edit /etc/rsyncd.conf
[bzr]
        comment = BZR source repositories
        path = /srv/bzr

bzr-daemon

apt-get install bzr

Install this xinetd file to start the daemon as needed.

service bzr-hpss
{
        port            = 4155
        type            = UNLISTED
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = nobody
        env             = HOME=/var/lib/bzr BZR_LOG=/var/lib/bzr/bzr-anon.log
        server          = /opt/savannah/bin/timeout-bzr-daemon
        server_args     = --no-plugins serve --client-timeout=3600 --inet --directory=/srv/bzr
        instances       = 40
}

The /opt/savannah/bin/timeout-bzr-daemon file.

#!/bin/sh
exec timeout --signal=SIGKILL 480m bzr "$@"

BZR Web

apt-get install loggerhead

FIXME: Document loggerhead and web.

Tidbit: Loggerhead is fragile and will sometimes die. Use monit to monitor loggerhead and restart it if needed.

check process loggerhead with pidfile /run/loggerhead.pid
  start program "/usr/sbin/service loggerhead start"
  stop program "/usr/sbin/service loggerhead stop"
  if failed port 8080 protocol http
    for 3 cycles
    then restart

Ensure robots.txt prevents crawling of every version ever.

...edit /opt/savannah/www/vcs/bzr/robots.txt
User-agent: *
Disallow: /

BZR Verification

For testing purposes all of the following needs to work. Where $USER represents a valid Savannah account name.

bzr branch bzr://bzr.savannah.nongnu.org/gnewsense/docs
bzr branch bzr+ssh://$USER@bzr.savannah.nongnu.org/gnewsense/docs

rsync rsync://bzr.sv.gnu.org/bzr/gnewsense/docs
rsync rsync://bzr.sv.nongnu.org/bzr/gnewsense/docs
rsync rsync://bzr.savannah.gnu.org/bzr/gnewsense/docs
rsync rsync://bzr.savannah.nongnu.org/bzr/gnewsense/docs

wget -O/dev/null http://bzr.savannah.gnu.org/lh/gnewsense/
wget -O/dev/null http://bzr.savannah.gnu.org/r/gnewsense/
wget -O/dev/null http://bzr.savannah.gnu.org/robots.txt

wget -O/dev/null https://bzr.savannah.gnu.org/lh/gnewsense/
wget -O/dev/null https://bzr.savannah.gnu.org/r/gnewsense/
wget -O/dev/null https://bzr.savannah.gnu.org/robots.txt

Subsystem CVS

CVS is a local source compile including locally developed patches.

FIXME: Document the CVS installation. Add CVS to the list of locally compiled software in need of special attention for security upgrades.

cvs rsync access

...edit /etc/rsyncd.conf
[cvs]
        comment = CVS source repositories
        path = /srv/cvs

CVS pserver

Install this xinetd file to start the daemon as needed.

FIXME: The implemented xinetd.d/cvspserver file needs cleaning before being documented here. The file listed here does not match the implemented file. The current file has many development comments in it.

service cvspserver
{
        socket_type             = stream
        wait                    = no
        user                    = nobody
        server                  = /opt/savannah/bin/timeout-cvs-daemon
        server_args             = -f --allow-root-regexp=^/srv/cvs/sources/.*$ --allow-root-regexp=^/srv/cvs/web/.*$ --allow-root-regexp=^/sources/.*$ --allow-root-regexp=^/web/.*$ --allow-root-regexp=^/cvsroot/[^/]+$ --allow-root-regexp=^/webcvs/[^/]+$ pserver
}

FIXME: Here too the implemented file does not match what is listed here. This is just the concept of it. The /opt/savannah/bin/timeout-cvs-daemon file.

#!/bin/sh
exec timeout --signal=SIGKILL 480m cvs2 -R "$@"

Savannah's web interface suggests /cvsroot rather than /sources for cvs access. Which means both are required to exist. Same for webcvs.

ln -s sources /cvsroot
ln -s webcvs /web

On the data host the repositories are located in /sources and /web but are also symlinked under /srv/cvs too.

ln -s net/vcs/sources /sources
ln -s net/vcs/srv /srv
ln -s net/vcs/web /web

ViewVC Setup

apt-get install apache2 viewvc

FIXME: Document ViewVC setup.

CVS Verification

For testing purposes all of the following needs to work. Where $USER represents a valid Savannah account name.

cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/cvsroot/hello co hello
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/hello co hello
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/srv/cvs/sources/hello co hello
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/srv/cvs/web/hello co hello
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/web/hello co hello
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/webcvs/hello co hello

cvs -z3 -d:ext:$USER@cvs.savannah.gnu.org:/cvsroot/hello co hello
cvs -z3 -d:ext:$USER@cvs.savannah.gnu.org:/sources/hello co hello
cvs -z3 -d:ext:$USER@cvs.savannah.gnu.org:/srv/cvs/sources/hello co hello
cvs -z3 -d:ext:$USER@cvs.savannah.gnu.org:/srv/cvs/web/hello co hello
cvs -z3 -d:ext:$USER@cvs.savannah.gnu.org:/web/hello co hello
cvs -z3 -d:ext:$USER@cvs.savannah.gnu.org:/webcvs/hello co hello

wget -O/dev/null http://cvs.savannah.gnu.org/
wget -O/dev/null http://cvs.savannah.gnu.org/robots.txt
wget -O/dev/null http://cvs.savannah.gnu.org/viewvc/

wget -O/dev/null https://cvs.savannah.gnu.org/
wget -O/dev/null https://cvs.savannah.gnu.org/robots.txt
wget -O/dev/null https://cvs.savannah.gnu.org/viewvc/

rsync rsync://cvs.sv.gnu.org/sources/hello/
rsync rsync://cvs.sv.nongnu.org/sources/hello/
rsync rsync://cvs.savannah.gnu.org/sources/hello/
rsync rsync://cvs.savannah.nongnu.org/sources/hello/

subsystem GIT

Rsync repository access

...edit /etc/rsyncd.conf
[git]
        comment = GIT source repositories
        path = /srv/git

git-daemon

apt-get install git git-man

Add this to the /etc/rc.local file. (Use the /etc/rc.local because init systems are rapidly changing from sysvinit to upstart to systemd and want to remain aloof from them all.)

# Start the git-daemon read-only git access daemon.
mkdir -p /run/git
chown nobody /run/git
(cd /run/git && env -i HOME=/run/git PATH=/usr/local/bin:/usr/bin:/bin su -s /bin/sh -c "git daemon --reuseaddr --max-connections=50 --init-timeout=10 --timeout=28800 --base-path=/srv/git --export-all --pid-file=/run/git/git-daemon.pid --detach" nobody)

GIT Web

apt-get install apache2
  ...a2enmod cgi # for cgit

FIXME: Document cgit and also gitweb.

GIT Verification

For testing purposes the following needs to work. Where $USER represents a valid Savannah account name.

git clone git://git.sv.gnu.org/hello.git
git clone git://git.savannah.gnu.org/hello.git
git clone git://git.sv.nongnu.org/hello.git
git clone git://git.savannah.nongnu.org/hello.git

fails: git clone --depth 1 git://git.sv.gnu.org/hello.git
fails: git clone --depth 1 git://git.savannah.gnu.org/hello.git
fails: git clone --depth 1 git://git.sv.nongnu.org/hello.git
fails: git clone --depth 1 git://git.savannah.nongnu.org/hello.git

git clone ssh://$USER@git.sv.gnu.org/srv/git/hello.git
git clone ssh://$USER@git.savannah.gnu.org/srv/git/hello.git
git clone ssh://$USER@git.sv.nongnu.org/srv/git/hello.git
git clone ssh://$USER@git.savannah.nongnu.org/srv/git/hello.git

git clone --depth 1 ssh://$USER@git.sv.gnu.org/srv/git/hello.git
git clone --depth 1 ssh://$USER@git.savannah.gnu.org/srv/git/hello.git
git clone --depth 1 ssh://$USER@git.sv.nongnu.org/srv/git/hello.git
git clone --depth 1 ssh://$USER@git.savannah.nongnu.org/srv/git/hello.git

wget -O/dev/null http://git.savannah.gnu.org/
wget -O/dev/null http://git.savannah.gnu.org/robots.txt
wget -O/dev/null http://git.savannah.gnu.org/cgit/
wget -O/dev/null http://git.savannah.gnu.org/gitweb/
wget -O/dev/null http://git.savannah.gnu.org/r/

wget -O/dev/null https://git.savannah.gnu.org/
wget -O/dev/null https://git.savannah.gnu.org/robots.txt
wget -O/dev/null https://git.savannah.gnu.org/cgit/
wget -O/dev/null https://git.savannah.gnu.org/gitweb/
wget -O/dev/null https://git.savannah.gnu.org/r/

rsync rsync://git.sv.gnu.org/git/hello.git/
rsync rsync://git.sv.nongnu.org/git/hello.git/
rsync rsync://git.savannah.gnu.org/git/hello.git/
rsync rsync://git.savannah.nongnu.org/git/hello.git/

Subsystem SVN

Rsync repository access

...edit /etc/rsyncd.conf
[svn]
        comment = SVN source repositories
        path = /srv/svn

svnserve daemon

Savannah is using ViewVC for Subversion repository web browsing.

apt-get install subversion

Install this xinetd file to start the daemon as needed.

service svn
{
        socket_type             = stream
        wait                    = no
        user                    = nobody
        server                  = /opt/savannah/bin/timeout-svn-daemon
        server_args             = -i -r /srv/svn/
}

Install this /opt/savannah/bin/timeout-svn-daemon file.

#!/bin/sh
exec timeout --signal=SIGKILL 480m svnserve "$@"

Apache setup for Subversion over https/http

apt-get install apache2 libapache2-mod-svn

We have assigned port 82 for Apache's subversion WebDAV setup. This port is added to /etc/apache2/ports.conf for Apache.

Ensure robots.txt prevents crawling of every version ever.

User-agent: *
Disallow: /

ViewVC Setup

apt-get install apache2 viewvc

FIXME: Document ViewVC setup.

Subversion svn Verification

For testing purposes the following needs to work. Where $USER represents a valid Savannah account name.

svn ls svn://svn.sv.gnu.org/administration/trunk
svn ls svn://svn.sv.nongnu.org/administration/trunk
svn ls svn://svn.savannah.gnu.org/administration/trunk
svn ls svn://svn.savannah.nongnu.org/administration/trunk

svn ls svn+ssh://$USER@svn.sv.gnu.org/administration/trunk
svn ls svn+ssh://$USER@svn.sv.nongnu.org/administration/trunk
svn ls svn+ssh://$USER@svn.savannah.gnu.org/administration/trunk
svn ls svn+ssh://$USER@svn.savannah.nongnu.org/administration/trunk

svn ls http://svn.sv.gnu.org/svn/administration/trunk
svn ls http://svn.sv.nongnu.org/svn/administration/trunk
svn ls http://svn.savannah.gnu.org/svn/administration/trunk
svn ls http://svn.savannah.nongnu.org/svn/administration/trunk

svn ls https://svn.sv.gnu.org/svn/administration/trunk
svn ls https://svn.sv.nongnu.org/svn/administration/trunk
svn ls https://svn.savannah.gnu.org/svn/administration/trunk
svn ls https://svn.savannah.nongnu.org/svn/administration/trunk

wget -O/dev/null http://svn.sv.gnu.org/viewvc/?root=administration
wget -O/dev/null http://svn.savannah.gnu.org/viewvc/?root=administration
wget -O/dev/null http://svn.sv.nongnu.org/viewvc/?root=administration
wget -O/dev/null http://svn.savannah.nongnu.org/viewvc/?root=administration

wget -O/dev/null https://svn.sv.gnu.org/viewvc/?root=administration
wget -O/dev/null https://svn.savannah.gnu.org/viewvc/?root=administration
wget -O/dev/null https://svn.sv.nongnu.org/viewvc/?root=administration
wget -O/dev/null https://svn.savannah.nongnu.org/viewvc/?root=administration

wget -O/dev/null http://svn.savannah.gnu.org/
wget -O/dev/null http://svn.savannah.gnu.org/robots.txt
wget -O/dev/null http://svn.savannah.gnu.org/viewvc/

wget -O/dev/null https://svn.savannah.gnu.org/
wget -O/dev/null https://svn.savannah.gnu.org/robots.txt
wget -O/dev/null https://svn.savannah.gnu.org/viewvc/

rsync rsync://svn.sv.gnu.org/svn/administration/
rsync rsync://svn.sv.nongnu.org/svn/administration/
rsync rsync://svn.savannah.gnu.org/svn/administration/
rsync rsync://svn.savannah.nongnu.org/svn/administration/

Host download0

Open questions:

Are people using download for a pserver? There are notes to that effect in the download /etc/default/iptables-rules file.

The download0 host NFS mounts the data from the NFS server. At the time of this writing the NFS server is the previous download host. The hope is that a new VM for data0 will be created to host the data. But there is no disk space for that VM. Therefore we are compromising.

mkdir /net /net/download
/etc/fstab contains
... download:/ /net/download nfs  defaults,async 0 0
echo 208.118.235.73 download.savannah.gnu.org download >> /etc/hosts
... above /etc/hosts entry works around boot ordering bug

Set up munin-node.

apt-get install munin-node
echo 'allow ^208\.118\.235\.77$' >> /etc/munin/munin-node.conf
service munin-node restart

File /etc/ssh/sshd_config
# Prevent endless of these messages being logged.
#   2017 Jan 14 21:52:48 vcs0 PAM service(sshd) ignoring max retries; 6 > 3
MaxAuthTries 3

Prevent ssh from passing through LANG and LC_* so as to avoid ungenerated locales on the local server from being seen by the perl script sv_membersh used for access control.

File /etc/ssh/sshd_config
#AcceptEnv LANG LC_*
...sshd defaults to not accepting any environment variables

TODO Items

A place for me to list things that I know about but haven't gotten to yet.

TODO: Set up /etc/ssh/ssh_known_hosts file.