Stuff I'm Up To

Technical Ramblings

Nginx SSL Certificate Error — October 14, 2020

Nginx SSL Certificate Error

We’re using client side certificates on an Nginx host to ensure the credentials of the connecting users and haven’t used the site for a while.

I tried to logon with a known good client certificate and know that nothing on the site config has changed and all I get in return is a 400 error with the message “SSL Certificate Error”, which is not at all helpful.

First I thought I’d regenerate my client certificate, no joy there. Still the same error. So I went through the process of verifying the CA cert matched my source and was still valid. Use openssl to verify my client certs, all looked good. Nothing I did allowed me to access the site unless I turned client certificate verification off.

ssl_verify_client off;

So what was my problem? I checked the nginx.conf and our logging was set to push out to a syslog server:

error_log syslog:server=logserver:515,severity=debug;

But I wasn’t seeing anything in the log about any errors. I checked the syntax of the config entry and found it to be missing the debug option – confusing I know, it looks like it should be using debug logging, but that’s just the severity setting for the syslog server, not Nginx. I added debug onto the end of the line:

error_log syslog:server=logserver:515,severity=debug debug;

Now I can see what the problem is with the certificates – there is no problem with the certificates, it’s a problem with my certificate revocation list being too old! I just need to regenerate and reissue a new one.

<190>Oct 14 20:38:22 proxy3.domain.tld nginx: 2020/10/14 20:38:22 [info] 12373#12373: *83301603 client SSL certificate verify error: (12:CRL has expired) while reading client request headers, client:, server: server.domain.tld, request: "GET /favicon.ico HTTP/1.1", host: "server.domain.tld", referrer: "https://server.domain.tld/web/database/selector"

By using openssl I can check the validity of my crl using:

$ openssl crl -in crl.pem -text
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = myCA
Last Update: Oct 14 19:40:03 2020 GMT
Next Update: Apr 12 19:40:03 2021 GMT

Now all I need to do is make sure I automate this process and dump a new crl onto the server at least every 6 months – I’ll probably do it monthly to be sure.


DRBD and LVM — October 13, 2020
iSCSI and Multipath —

iSCSI and Multipath

If you’ve installed the open-iscsi and multipath-tools you might still find that you multipaths aren’t working.

sudo apt install open-iscsi multipath-tools
$ sudo lsblk
sda 8:0 0 50G 0 disk /mnt/sda
sdb 8:16 0 40G 0 disk /mnt/sdb
sdc 8:32 0 50G 0 disk
sdd 8:48 0 40G 0 disk

I ended up with the disk devices sda, sdb, sdc and sdd but no multipath.

Continue reading
Docker IP Address Error — October 2, 2020

Docker IP Address Error

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

We have a staging environment that runs a lot of container sets for customer testing and recently I went to add a new container set ad got the above error.

On my own system I saw this kind of issue when I ran out of IP pool space because it overlapped with the VPN I was using. I solved it by creating another docker network and using an override to add it to my container set. I didn’t want to do the same here, but testing it out by adding the network revealed I was out of IP pool space.

Continue reading
Installing Ansible AWX — September 14, 2020

Installing Ansible AWX

Starting to look at a nice front end for Ansible and found a few quirks with the process that need to be taken care of.

The actual installation of AWX needs Ansible installing on the system you’re installing from. As I’m installing from localhost to localhost this means I need Ansible installed locally. Sounds obvious, but it’s not in the prerequisites. You need to install, docker, docker-compose, the docker-compose python module AND ansible.

Continue reading
PowerDNS and ISC DHCPD — September 8, 2020


As we host lots of sites we have lot’s of DNS. Currently it’s a database driven old version of ISC BIND. There are certain tasks we need to do manually to add records like SPF and TXT and there is no dynamic update. Our main aims are to have a easier means of our support staff updating domains with all the modern records for DMARC and DKIM, but also lets us use dynamic entries to help with the Let’s Encrypt certificate creation process.

We have a strategy that all our deployments begin as virtual first to keep things as mobile and as resilient as possible. This means Docker or KVM, before we look to deploying natively. This solution is going to be within a Docker container set.

Continue reading
Performance Monitoring — September 2, 2020
Humidity and Temperature Monitoring — August 16, 2020

Humidity and Temperature Monitoring

With the temperature the past few weeks I thought I’d setup a means of monitoring it to trigger alerts in Grafana or Incinga2 by using a Raspberry Pi to collect the data.

There are projects for this all over the place so I thought I’d take one and expand upon it –

The original work records data to a CSV file which we can’t use for Grafana. We need something a little more enterprise, like MySQL or PostgreSQL. Which is a shame because for this sqlite would probably have been good enough.

Continue reading
Libc6 and OpenSSH — August 14, 2020

Libc6 and OpenSSH

This morning was pretty much written off by some nasty occurrence I didn’t pay attention to yesterday.

Whilst trying to fix gimp with lots of apt manipulation I saw a message saying that mention that openssh-sftp-server was no longer required and could be removed using autoremove. I should have checked that out. Because today when I tried to logon to my PC remotely there was no ssh!

Luckily a fellow admin was in the office and able to try to reinstall openssh-server for me, only to be presented with a message stating that libc6 was preventing the version of openssh-server from being installed.

libc6 : Breaks: openssh-server (< 1:8.1p1-5) but 1:7.9p1-10+deb10u2 is to be installed
Continue reading
Raspbian & Realtek 8192eu Wifi – Revisited — August 8, 2020

Raspbian & Realtek 8192eu Wifi – Revisited


It’s been a while since I originally wrote about the Raspbian & Realtek 8192eu WiFi USB Wifi adapter and just recently I wanted to build something from some old Pi’s I had knocking around. I needed Wifi so went to install the adapter I have using my notes. They no longer apply. Seems the creator of the previous builds has retired them and a new method is needed.

This lead me here:

Only trouble is with my kernel 4.19 it wasn’t able to get the headers necessary to build it.

Let’s really upgrade things. It’s an old Pi B+ I’m using, no wifi – hence the need for a USB dongle. Let’s use rpi-update:

sudo rpi-update

This took it from:

Linux raspberrypi 4.19.118+ #1311 Mon Apr 27 14:16:15 BST 2020 armv6l


$ uname -a
Linux raspberrypi 5.4.51+ #1332 Tue Aug 4 18:28:38 BST 2020 armv6l GNU/Linux

Now when I follow the setup from github, it starts making:

$ sudo dkms install rtl8192eu/1.0

Kernel preparation unnecessary for this kernel. Skipping…

Building module:
cleaning build area…
'make' all KVER=5.4.51+………………………………………………………………………………………………………………………….

… Some considerable time later.

cleaning build area…
DKMS: build completed.
Running module version sanity check.
Original module
No original module exists within this kernel
Installing to /lib/modules/5.4.51+/kernel/drivers/net/wireless//
Warning: Unable to find an initial ram disk that I know how to handle.
Will not try to make an initrd.
DKMS: install completed.

This is where it all came to a crashing end for me.

modprobe: ERROR: could not insert '8192eu': Exec format error

The module still fails to load and there are complaints in dmesg etc. looking like:

8192eu: disagrees about version of symbol module_layout
could not insert module 8192eu.mod: Invalid module format

So I’ve given up for now. I’ve seen others spend hours messing with kernel version mismatches and may need to get my head around that some more, but for now I just plugged in another working USB Wifi adapter.

History to Timesheets — July 29, 2020

History to Timesheets

I’m not great at remembering what I did so I can include it on my time logger. I found I could trawl through my zsh history file to get an idea of what I’d been up to and use it as a memory jogger to go back in time and update my timesheet.

By using a mixture of sed and awk I was able to grab the columns from the ~/.zsh_history file. I can import this into a spreadsheet and then use a formula to convert the epoch date format to a proper date/time I can read.

Continue reading
Apache Directory Studio – JNDI — July 28, 2020

Apache Directory Studio – JNDI

I’ve never had a comfortable relationship with Java. Every time something goes a bit wrong in something that uses Java, I spend hours and even days trying to figure out why the wheels have come off.

Given that Apache Directory Studio hasn’t been updated in years, when all of a sudden my remote connection to manage my LDAP server stopped working – I had to think, “it’s Java.”

I wasn’t wrong

Continue reading