Previously, I had a project to allow running the Cobalt OS (almost) unmodified in a VM. Mostly, this came about due to being nerdsniped into it. I do also think VMs are an accessible way for people to experiment with legacy systems - it's rarely necessary to have access to physical hardware in order to get a feel for a system.
Shortly after completing the VM project, I got my hands on a physical Qube 3, which naturally means trying to go entirely the other way - can a machine released in 2002 (22 years old at time of writing) still provide services on the modern Internet?
Out the box, there is a webserver, Perl and PHP. However, the versions provided are very old. Using these directly would be possible, however there is such a long history of vulnerabilities in the shipped software that it feels like an unwise foundation for trying to run public services.
I did also attempt to bootstrap a more modern userland, with a view to having a more modern machine act as a proxy. This approach became impractical, as everything required compiling (from GCC up) and the cycle time was too great for practical experimentation. In early tests, building GCC 4.x was taking over 25 hours per attempt. I also met with mixed success trying to do the builds in a VM - they were significantly faster, but it proved challenging to ensure autotools didn't tune the build for a more modern CPU.
It may be possible to explore this further with a more leaning on cross-compiling, or by using something more modern to help bootstrap the toolchain.
After the failure to bootstrap from Cobalt OS, I did a little more reading and found a blog post about running alternative OSes on the Qube 3. This documented running Slackware 14.2 on a Qube 3. Slackware 14 is relatively recent, coming from 2016, and even more excitingly Slackware 15 (current as of writing!) is still built for i586, so at least the userland could potentially run.
Following the instructions in the blog, I successfully got Slackware
misunderstanding the instructions and creating a /boot partition. Once these issues were solved, everything just worked.
Slackware 14 is still pretty old - far too old for just lazily git cloning software and expecting it to compile. Can we keep pushing the Qube until it's running a userland that's still getting security patches? Bringing the kernel up to date is currently out of scope, as it will require porting patches to some pretty low-level C. For the purposes of this, an old-ish kernel is likely fine anyway - most of the spicy modern exploits are Intel's fault, and the Qube 3 CPU is an AMD chip from before the days of branch prediction.
As it turns out, yes we can!
The key first step was to flash the Cobalt ROM. I did this from the stock Cobalt OS, which I installed by using my existing Qube OS on a VM project, but using a USB CompactFlash adaptor as the disk.
Once I had an OS I could telnet to, I followed the steps from the same blog to flash a ROM from the Cobalt ROM project.
I the confirmed the update had worked correctly by watching the boot process via a serial terminal.
The next step was to install Slackware 15 to a CompactFlash card using a VM for ease. I made sure to have all the CLI/shell utilities, removed GUI/desktop sections, and made sure to install the "libraries" section (since I plan to build software). The mistake I made on the first attempt was creating a /boot partition - had been reading lots of old documentation as part of this project, and internalised a half-remembered thing about older systems not being able to boot from large partitions. This is not the case on the Qube - the ROM will only look for a kernel in /boot on /dev/hda1.
While partitioning, I also created a far larger swap partition than I usually would for the size of physical RAM available. In the case of this machine, there's 32MB of RAM, and 1.5GB of swap. This is hugely excessive, but when I was setting up the machine, I had no idea how much it might need to use during things like compiling. With hindsight, 100MB is way closer to what's needed.
Once the Slackware install finished, I built a Linux kernel based on St John Karp's guide and installed it, then made sure to change fstab and inittab as noted so disks and consoles would run. I also created a user at this point, so it was available later.
Then, I moved the SD card to the machine, plugged in a serial terminal, and was surprised to see things work.
I was pretty keen to try and run "a service", rather than doing something like hosting this static site on it. The obvious candidate was a Fediverse server of some kind - it's an actual service, but pretty much everyone apart from me would be using it indirectly so it being slow wouldn't be an issue. Someone else's server would queue requests and handle retries for me, it falling over for a few minutes is fine.
For my general wittering online, I use Mastodon. There's no way that will run nicely on a machine of this vintage - the resource requirements are just too high (my single-user instance seems to use around 2GB of RAM just to exist).
Luckily however, I'd recently come across Snac2 (Social Networks Are Crap). This is a minimalist ActivtyPub server written in C, with far more modest resource requirements.
First, I made a user for Snac, then became that user, cloned the repo and checked out the latest stable release with git:
git clone https://codeberg.org/grunfink/snac2.git
cd snac2
git checkout 2.54 # Latest at time of writing
The only gotcha with building on the Qube 3 was needing to disable shared memory, due to lack of CPU support.
make CFLAGS=-DWITHOUT_SHM
Once snac has built, need to initialise the storage directory and create a user:
./snac init /home/snac2/data
./snac adduser /home/snac2/data theothertom
I mostly copied another init script, and didn't bother with a pidfile etc.
$ cat /etc/rc.d/rc.snac
#!/bin/sh
#
# /etc/rc.d/rc.snac
#
# Start/stop/restart/graceful[ly restart]/graceful[ly]-stop snac
#
# To make snac start automatically at boot, make this
# file executable: chmod 755 /etc/rc.d/rc.snac
#
case "$1" in
'start')
su snac2 -c '/home/snac2/snac2/snac httpd /home/snac2/data/' /var/log/snac.log
;;
'stop')
pkill snac
;;
*)
echo "Usage: $0 {start|stop}"
;;
esac
That needs to be marked as executable to start automatically on boot.
Snac requires a frontend proxy to listen for incoming requests. Since Slackware package Apache by default, it seemed like a reasonable choice.
First, enable HTTPS (in /etc/httpd/extra/httpd-ssl.conf). A minimal config looks like this:
Listen 443
SSLCipherSuite HIGH⭐!MD5:!RC4:!3DES
SSLProxyCipherSuite HIGH⭐!MD5:!RC4:!3DES
SSLHonorCipherOrder on
SSLProtocol all -SSLv3
SSLProxyProtocol all -SSLv3
SSLPassPhraseDialog builtin
SSLSessionCache "shmcb:/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout 300
<VirtualHost _default_:443>
DocumentRoot "/srv/httpd/htdocs"
ServerName social.cobaltqu.be:443
ServerAdmin tom@deathbycomputers.co.uk
ErrorLog "/var/log/httpd/error_log"
TransferLog "/var/log/httpd/access_log"
SSLEngine on
SSLCertificateFile "/etc/httpd/social.cobaltqu.be.cer"
SSLCertificateKeyFile "/etc/httpd/social.cobaltqu.be.key"
SSLCertificateChainFile "/etc/httpd/server-ca.crt"
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/srv/httpd/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog "/var/log/httpd/ssl_request_log" \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b"
This config was heavily based on the provided defaults. Most of the change was providing a path to certificates created by ACME.sh.
I created a little script for use in cron, then set it to run at a random time each day:
# cat /usr/local/bin/renew.sh
#!/bin/bash
set -uo pipefail
/root/.acme.sh/acme.sh --issue -d social.cobaltqu.be --server letsencrypt -w /var/www/htdocs/
cp /root/.acme.sh/social.cobaltqu.be_ecc/ca.cer /etc/httpd/server-ca.crt
cp /root/.acme.sh/social.cobaltqu.be_ecc/social.cobaltqu.be.cer /etc/httpd/
cp /root/.acme.sh/social.cobaltqu.be_ecc/social.cobaltqu.be.key /etc/httpd
apachectl restart
Something that turned out to be key to maximising availability was limiting concurrent requests to Snac as much as possible. Each request can take hundreds of milliseconds to process, which on top of doing modern TLS can end up being close to a second of CPU time per request (there will often be images too, which can also be quite heavy to serve).
I did add some rewrite rules in an attempt to get Apache to serve images directly. This doesn't make a huge difference, but it does help a bit. If nothing else, it avoids tying up a Snac thread.
# cat /etc/httpd/extra/snac2.conf
#Allow ACME to renew certs
ProxyPass /.well-known/acme-challenge !
#ProxyPassMatch "^/(.*)/s#/" !
RewriteEngine on
# The following turns requests like this:
# https://social.cobaltqu.be/theothertom/s/829b2d11f36149ba00e226393016bef2.jpeg
# into files Apache can serve directly like this:
# /home/snac2/data/user/theothertom/static/829b2d11f36149ba00e226393016bef2.jpeg
RewriteRule ^(.*)/s/(.*) /home/snac2/data/user/$1/static/$2
# max=1 limits Apache to a single backend request per child.
ProxyPass / http://localhost:8001/ max=1
ProxyPassReverse / http://localhost:8001/
ProxyPreserveHost on
I found that it helps a little to tune the Apache MPM (Multi-processing model). The "event" MPM feels like the most lightweight option for a scenario where there may be a large number of requests close together. Can also tune it to limit the number of requests than can be in-flight at one time:
# /etc/httpd/extra/httpd-mpm.conf
<IfModule mpm_event_module>
StartServers 1
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 25
MaxRequestWorkers 3
MaxConnectionsPerChild 0
</IfModule>
# /etc/httpd/httpd.conf
# There will be LoadModule lines already in the file, (un)commenting
# existing lines will be easier to work with than copy/pasting.
LoadModule mpm_event_module lib/httpd/modules/mod_mpm_event.so
#LoadModule mpm_prefork_module lib/httpd/modules/mod_mpm_prefork.so
#LoadModule mpm_worker_module lib/httpd/modules/mod_mpm_worker.so
LoadModule rewrite_module lib/httpd/modules/mod_rewrite.so
LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
LoadModule ssl_module lib/httpd/modules/mod_ssl.so
LoadModule rewrite_module lib/httpd/modules/mod_rewrite.so
Include /etc/httpd/extra/snac2.conf
# Can also disable some other modules, unless needed for something else:
#LoadModule authn_core_module lib/httpd/modules/mod_authn_core.so
#LoadModule authz_groupfile_module lib/httpd/modules/mod_authz_groupfile.so
#LoadModule authz_user_module lib/httpd/modules/mod_authz_user.so
#LoadModule auth_basic_module lib/httpd/modules/mod_auth_basic.so
#LoadModule autoindex_module lib/httpd/modules/mod_autoindex.so
Trying to limit the number of requests that are in-flight does still lead to 503 errors, but it minimises the backlog, so at least earlier requests can succeed - other ActivityPub servers will retry and eventually cache content.
After all the config was in place, I rebooted the machine to double-check that the startup config was correct.
Snac is relatively simple, so you don't get handy things like automatic image compression. If you share images, pre-compressing them makes a big difference.