Debugging with docker is delightful

I run most of my web services in containers though I have sometimes wondered why – or rather, if it was really necessary as a hobbyist who just has one of each application and rarely has to move or replicate containers. Like coding an intricate and beautiful class in a programming language that you then proceed to make one instance of. Ever. Yes, it might be inherently more correct but… did you need to and at what cost?

An answer of sort came recently when I was trying to debug some applications. The ease with which I could switch the images I was using to build on made it a breeze to test my theories as to what was the issue. There is nothing new or groundbreaking about it, it was just some very pleasant experiences, so I thought I would share.

Syncthing’s rotten in the state of Ubuntu

Syncthing is in the Ubuntu repos so I proceeded to install from there when I was setting my desktop up with Ubuntu 20.04, a clean install over old 18.04. I have a few syncthings talking to another on my local network – a laptop here, a raspberry pi there, and the desktop on top of that – that have different versions from different sources (Fedora, Debian and Ubuntu) but since Syncthing hit 1.0 I haven’t had any issues with compatibility.

This time was different, though. My Fedora laptop’s instance was complaining about the shiny new Ubuntu one being older than dirt:

May 19 20:05:01 PINCIAN syncthing[959]: [I734U] WARNING: Not sending symlinks to old client THMLPCB-6SVWMPS-W24T67U-NXJAZFG-ZZ3ZQTH-2XMIZM5-S4DBRCA-2B2JUQO - please upgrade to v0.14.14 or newer

A look in the logs confirmed the laptop’s misgivings. Apparently the Ubuntu version had turned 50 a couple of months ago:

[start] INFO: Generating ECDSA key and certificate for syncthing...                                                                             
[O3SM6] INFO: syncthing unknown-dev "Erbium Earthworm" (go1.13.5 linux-amd64) unknown@unknown 1970-01-01 00:00:00 UTC                           
[O3SM6] INFO: My ID: THMLPCB-6SVWMPS-W24T67U-NXJAZFG-ZZ3ZQTH-2XMIZM5-S4DBRCA-2B2JUQO

More than that, the version number clearly hadn’t been set properly and wasn’t being communicated to the other clients. I was pretty sure this had to be a bug but I wanted to see if I could replicate it before sounding off.

To avoid messing up my new desktop install I turned to docker and whipped up my own syncthing image based on Ubuntu 20.04. Sure enough the same “unknown-dev” version number was there. Now that I had gone to the trouble of writing the Docker file etc. I thought I might as well do some more tests. Was I wrong that this hadn’t been an issue on my old desktop install? Was it something I was doing wrong in configuration? Had it been a longstanding issue? Did it come from Debian? All these questions could now be answered easily (spoiler: No, no, yes, no-or-they-have-fixed-it-now):

ubuntu:18.04
77syncthing  | [start] INFO: Generating ECDSA key and certificate for syncthing...                                                                             
77syncthing  | [FVZSE] INFO: syncthing v0.14.43-ds1 "Dysprosium Dragonfly" (go1.9.3 linux-amd64) debian@debian 2018-02-06 04:01:09 UTC [noupgrade]             
77syncthing  | [FVZSE] INFO: My ID: FVZSECH-6XQLY6U-4GENWJO-4VI35S6-E3CATNI-AX2JJ2N-T77AZ23-5VI43AT   

ubuntu:19.10
77syncthing  | [start] INFO: Generating ECDSA key and certificate for syncthing...                                                                             
77syncthing  | [5JS67] INFO: syncthing unknown-dev "Erbium Earthworm" (go1.12.9 linux-amd64) unknown@unknown 1970-01-01 00:00:00 UTC                           
77syncthing  | [5JS67] INFO: My ID: 5JS67BR-H5NWDCN-2R5I36K-OTWYUDU-CNWPAQJ-7CGC2NV-LRYZ6OT-B2SR3AR 

debian:sid (20200514)
77syncthing  | [start] INFO: Generating ECDSA key and certificate for syncthing...                                                                             
77syncthing  | [Y2LKO] INFO: syncthing v1.1.4-ds1 "Erbium Earthworm" (go1.14 linux-amd64) debian@debian 2020-03-13 00:59:32 UTC                                
77syncthing  | [Y2LKO] INFO: My ID: Y2LKOYL-LPWTJSB-5ZXOMNZ-3NZGRLT-4EMNAPG-EG2I47P-JOEWAGV-WDBSNQ3 

I have been listening to the Ubuntu Podcast recently and taken Alan’s anti-distrohopper talk to heart so I went proudly in search of a place to report this bug… only to find that it had been reported a long time ago, when the issue was introduced and the workaround simply was to use syncthing’s own repositories.

Imagick there’s no official image

The new WordPress Site Health tool told me my site’s health was good… but it had some issues, it wanted me to look at. Among them was that I was missing the (optional) php extension “imagick”. Which I thought was strange because I was sure it was included in my docker image for php-fpm. Turns out it wasn’t, probably due to issues to do with building the image.

Some quick background: I am using a php-fpm container as a standalone php processor as is customary with Nginx and, says Jim Salter, increasingly with Apache. Php extensions are added to the php-fpm image by building my own image on top of one of the official one from Docker hub. My image pulls a Debian-based hub image, installs some libraries from the Debian repositories and installs the php extensions using the docker tool, docker-php-ext-install (see the docker hub page for the php image for details).

After some searching I found a reasonably authoritative-looking suggestion on how to install imagick – apparently the standard docker-php-ext-install route doesn’t quite work for imagick – and proceeded… straight in to a wall of

No releases available for package "pecl.php.net/imagick"
install failed

The php:fpm tag is currently synonymous with php 7.3. From an obscure thread on the WordPress forums about imagick’s requirements and compatibility I got the idea that maybe the php version was the issue. Changing FROM php:fpm to FROM php:7.2-fpm and the issue went away! To be replaced by another one! Now it was the gd extension that had problems.

To cut a long story short, the gd extension needed one set of configuration commands for versions of php prior to 7.3 and another set for 7.3 and onwards. To confirm these dual issues – not to mention actually building an image that I could use and would satisfy that nagging Health Status notification – I ended up “crossing” two base images (7.2-fpm and 7.4-fpm) with two versions of my Dockerfile (one old-gd-style, one new-gd-style). Since I got the results I expected – 7.2-fpm worked with old style, 7.4-fpm worked with new style and the two other combinations failed (no need to test 7.3 as imagick had ruled that out) – I reasoned that I had figured it out. And since WordPress sniffed at my 7.2 based image (due to support ending in 6 months’ time), I also had just one result I could use.

I don’t know if any of this justifies the time and effort I have spent trying to understand containers and the issues they produce, but then you could say that for any of these projects! 🙂 I certainly couldn’t have debugged these issues half as well (I would have run out of patience a long time before) – and probably not fixed at least the latter – had I not containerised the applications.

To end on a complete non sequitur: After spending 15 minutes searching on flickr, I made the design choice to not include yet another random image of ship containers as the header. You’re welcome 🙂

Leave a Reply