GB4LHS Special Event Station

Over the weekend, I operated a special event station, GB4LHS to celebrate the London Hackspace reopening in Wembley, and also to demonstrate amateur radio to visitors at the open day. Some things worked really well, other parts of the setup could really have done with improvement. In this post I hope to describe the details of this.

Some Statistics

  • Total QSOs: 76
    • FT8: 68
    • SSB: 8
  • By Band:
    • 17m: 20
    • 20m: 20
    • 40m: 36
  • Countries Worked: 36
    • Most Contacted: Federal Republic of Germany (13 QSOs)
  • Time between first and last QSO: 23 hours

I have actually operated a special event station once before – I organised GB50SWN back in 2007 for JOTA. At the time I only had a foundation license, however with the (mostly quite distant) supervision by the full license holder, I made a total of about 4 contacts over two days. I comfortably beat that “record” at least! I’ll admit I actually set myself a slightly higher target for this event of 100 QSOs – but I’ll take 76 as close enough. Now I think about it, I might actually have got a few more if I had managed to get set up earlier and therefore got more sleep the night before, hence probably started again earlier.

I was probably most impressed by the 40m FT8 contact into Australia, and the 40m SSB contact into Kazakhstan. I was putting out 30W on FT8 and 100W on SSB. What’s interesting is both of these would have been towards the end of the dipole antenna – and therefore not in the most sensitive part of its radiation pattern,

The Setup

I decided to construct a variant on a fan dipole for this event, supported by the building at one end and a telescopic pole at the other. Google Earth suggested I had about 22m to play with – so enough space for a 40m band dipole. This deserves a whole section to itself, so I’ll discuss that more later.

I fed this with my Yaesu FT-991A – I decided to use this rather than the Hackspace club’s FT-450 because it has a more modern receiver, and also because it has a built in voice recorder, so I could record CQ calls and play them back, saving having to speak so much. The FT-991A also has a built in sound card for data modes, however I ended up not using this due to incompatibilities with the computer I was using. Instead I used a MiniProSC I had with me in case I decided to use the FT-450.

In order to look more impressive, be easier to work with, and in case there was an opportunity to send greetings messages, I decided to use a microphone mounted on an inexpensive boom arm, and a keyboard pedal for push to talk. I fairly hastily built a cable to wire this into the Yaesu 8-pin microphone socket – an easy task as long as you realise Yaesu call pin 1 what I would call pin 8 on the 8P8C connector. The microphone I chose was the relatively inexpensive AKG P3S – I have always liked AKG audio equipment and this did not disappoint – I received compliments on my audio quality from all the SSB contacts I made. I also used AKG K240 MKII headphones for some contacts. I have had these headphones for a few years now and they’re fantastic – I can wear them for hours without them getting uncomfortable, and they are well built (I can be quite hard on my equipment!) with a removable cable.

I did try using an external speaker, but it turned out to be worse than the one in the FT-991A, so ended up using a mixture of the built in speaker and headphones.

Live Website and Logging

Live Status website

For logging, I used Cloudlog written by 2M0SQL – I spent quite a bit of time before the event fixing various bugs I hit in my setup. Overall though, it was a very useful tool and mostly worked well.

In addition to this, I had set up a live log website, with some auto refreshing (using JSON XHR requests) stats such as current frequency and the last 10 contacts logged. The idea was it would make it easier for people to catch me on the dial. Although this was hastily put together, it worked well though I don’t know how much use it got. I can see myself reusing this code in future. It was a flask app on top of the cloudlog database.

What Went Well (The Rig)

XLR to 8-pin microphone plug. I didn’t have any line sockets for 6.35mm jacks, so put a panel socket in a box.

The overall setup was quite successful, with plenty of contacts made, and plenty of compliments on audio quality. The one thing I wasn’t entirely happy with was the receive audio – I really needed headphones to hear the often weak or noisy signals, but I was reluctant to use them during the day when people were around because then it wouldn’t have been much good as a demo. I’ll mention some ideas I have for improvements here later in the post.

Once I figured it out, the 6m telescopic fibreglass pole worked well to support one end of the antenna, and the other end was tied to a conveniently located eye bolt inside a window.

What Didn’t Work So Well (The Antenna)

An early-ish iteration of the feedpoint. Note the ferrule used to keep the tension on the crimp connector low, and also the bends in the acrylic for strength.

I had the idea of making a fan dipole with laser cut separators for the 40, 20 and 17m bands. A fan dipole is a brilliantly simple, yet effective antenna. Essentially, you build multiple dipoles for different bands, and connect them all to the same feed point. The radio waves tend to use the most resonant wire (I guess because it’s the lowest impedance). I chose this design because I didn’t want the inefficiency or hassle of an ATU. Further, previous experiments with custard proved unsuccessful on the HF bands so that wasn’t an option. The idea of the laser cut separators was to keep the wires fairly close together, to keep the ends as high as possible.

This worked better in my head than it did in reality. The initial design for the separators had a tendency to snap under the tension of the antenna. This was resolved by bending the ones at the ends to increase strength. Initially, I planned to linear load the 40m element to shorten the antenna slightly. As it turned out, this just complicated things, and I really didn’t need to. After much trial and error (mostly error), I got it nicely tuned on three bands. One thing that really helped here was an idea from fellow Hackspace member Simon to put the wires on separate crimp connectors, allowing easy disconnection of the wires individually for tuning, then reconnection for final adjustment and operation.

It tuned! Or did it?

However once up in the air, and connected to the intended coax run, the resonance on all bands dropped, making it rather less good (except on 17m where it improved if anything):

Results after bringing the coax inside. I just realised 18.1MHz isn’t here, but it was actually pretty close to 1.0:1.

I think there are a few things going on here. Firstly, I suspect the wires twisted a bit, and probably got closer together. Secondly, I had attached a longer piece of coax, with chokes on which were not present on the piece I used for testing. Thirdly, the coax was coming off at a bit of an angle, which probably tweaked things a bit.

In the end, I decided to run with it. I figured the rig ATU could probably make up the rest, and in any case, a 3:1 SWR ratio isn’t completely terrible. The rig seemed happy enough on most frequencies, so I wasn’t too bothered. The only downside was SWR got a bit too high on the upper part of the 20m band. I ended up cutting that element from slightly thinner wire, and I believe thinner wire tends to have a narrower bandwidth, so the SWR went up quite sharply above about 14.200. It was usable on the lower part of the band for FT8 though.

As it turned out, the antenna was most resonant on the 17m band, and the built in ATU, while effective on 40m, actually made things worse on 20m! In any case, radio waves got out. In future, I think a more traditional fan dipole with rope or twine between the elements might be more effective – I believe the close spacing of my laser cut spacers made it too sensitive when tuning and too prone to twisting.

One thing I definitely underestimated with this was the amount of tension in the top element – and therefore the pole and some of the spacers. In the end, the top of the pole ended up (mostly accidentally) tucked behind the lamp on top of the lamp post, which provided extra support and stopped the pole from bending nearly as much as it otherwise would have done. It took a considerable amount of force to pull the top of the antenna level-ish – I’m not sure if this is inherent in fan dipoles or if it’s down to my laser cut spacers – but it was definitely more than I expected.

The antenna, strung between the building (via an open window) and a fibreglass pole tied to the lamp post

In the end, I could easily have got away with the ends being lower down, so I really didn’t need the extra complexity of the laser cut spacers. That said, they did give it a more Hackspace look!

One other thing not antenna-related that didn’t go so well was the CAT control interface. As well as having to use an RS-232 cable for this, I also found rigctld regularly stopped responding and timing out with the rig, or exited. This meant I had to restart it more than I’d like. rigctld was being used by both WSJT-X for FT8 and also a script updating cloudlog. A few other club members told me they have seen similar issues on that machine, so it’s likely it will get a fresh OS install in the near future.

Things To Try Next Time

Apart from tuning the antenna better, the biggest thing I would improve is the receive audio setup. Next time I do something like this, I want to have some decent speakers hooked up to the rig, but with an audio splitter so I can still use headphones to get some isolation and hear the weaker signals. One of the biggest problems I had was people having (loud) conversations right outside the (open) door which did not make the high RF noise level any easier to work with. This way, visitors can still hear what’s going on, while the operator can actually hear the more difficult signals.

I had a DMR rig handy, but didn’t really do much with it, so apart from showing off a pi-based hotspot a bit, I could do more with that.

One idea I had but decided against on this occasion – mostly to avoid scaring visitors off – is live streaming the audio from the transceiver and maybe even some video. Video probably isn’t the greatest plan since people are probably reluctant to go anywhere near a live video camera, but audio streaming might be fun.

I’m not sure it really counts as an improvement, but I’m told I need to have a custard antenna at the next open day. I guess time will tell if I actually do that or not.


Overall it was a very fun day, and I’m keen to do it again. A few tweaks (notably a setup allowing me to actually hear the signals) would make a big difference. I was, as ever, pleased with the performance of the Yaesu FT-991A, and the voice memory was indeed very handy for easily calling CQ.

The antenna wasn’t as good as I hoped, but it still got signals out, and I learnt a lot I can apply in future to make it less of a headache, so overall I guess that’s a good thing.

Introducing: The Custard Antenna

In a previous post, I loaded up a bowl of custard with an ATU. I called the post “Custard Antenna” however, really, it should probably have been titled “Custard dummy load” – with both antenna wires immersed in the bowl, the custard was no doubt mostly behaving as a resistor, with any pickup or radiation likely to be on the wires – though removing the wires did also prevent it from receiving, which does make me think it might be behaving as a somewhat lossy loop antenna. I chose this format not because I thought it would work as an antenna, but because the whole thing was inspired by the phrase “tune a bowl of custard” so I wanted to take it literally.

Anyway, this post is about making an actual custard antenna. In truth, when I started this I wasn’t actually sure how to do it. I thought about using guttering as troughs but it would be awkward, especially since I couldn’t quite think of a way to elevate the antenna to avoid the power simply coupling into the ground.

The Plan

The design I settled on was clear plastic tubing, with an inside diameter of 12mm – I figured this is wide enough that the fairly thick antenna fluid custard can be poured down it, but small enough that it won’t be so heavy that it’s impossible to get up in the air.

Taking some rough measurements from a box of antenna fluid custard, I calculated the density to be 1094Kg/m^3 – just a bit more dense than water – which makes sense. This means 5m of 12mm tubing would contain about 600g of antenna fluid custard – which I think should be light enough to be supported by my portable antenna pole. This is a good start!

My plan was to make a vertical for the 20m band using the tube of antenna fluid custard. This leaves the question of what to make the radials out of. Custard could also be used for the radials, and this would make it very much a custard antenna. On the other hand, using regular wire radials would allow swapping a wire element out for the custard for direct comparisons – I decided on the latter.

Interestingly, someone on Twitter pointed out that water actually has a very low velocity factor – in the order of 10% – and pointed out custard could be similar. Velocity factor is a measure of how fast electromagnetic waves move down a conductor or waveguide. It is quoted as a fraction or percentage of the speed of light in a vacuum. Typical values might be 70% for coaxial cable or 95% for balanced feeder. This matters when building an antenna, since a 1/4 wave element needs to be 1/4 of the wavelength in the conductor, not 1/4 of the wavelength in free space in a vacuum (which is how the wavelengths are typically quoted). Practically, if true, this could result in a very short (and probably very lossy) antenna for the wavelength.

Preliminary Experiments

One interesting question would be how different variants of custard compare, electrically. I already have 3kg of custard left over from a previous experiment, so unless any others are significantly better, that’s most likely what I’ll use. It’s still worth testing other variants though. For these experiments, I used tins of custard, since the range was greatest.

Tesco Custard

This is what I used last time to tune a bowl of custard, and have 3kg of left. It’s the baseline to compare to.

Tesco Low Fat Custard

This is the low fat variant of the baseline. It is actually higher in sugar which could make things interesting and potentially behave slightly differently.

Stockwell Custard

This is lower in sugar and fat than the baseline, and also cheaper. Again, it may behave differently.


With the exception of the condensed milk, which seemed to have relatively high resistance (measured using antenna analyser placed across a 1M long tube of the substance under test), the custards all behaved similarly electrically. Some exhibited more yellowness than others (definitely an important parameter). Resonances seemed to broadly match what I’d expect for wire of the same length, which somewhat throws out the velocity factor idea (I’ve since learnt velocity factor has more to do with the insulator than it does the conductor so this actually makes sense)

Condensed milk tended to leak out the end of the tube where the wire entered, which was not a problem with the rather thicker custard.

Going Full Scale

The time came to build a full size custard antenna. I took 5m of the plastic tubing, placed on end into a box of antenna fluid custard, and started sucking on the tube, like a huge straw. At first, it seemed to be working well. The problem was, the more custard entered the tube, the harder it became to draw the antenna fluid custard up the tube. In the end, I got about 2.2m of custard into the tube. I decided this was going to have to be enough, and proceeded to attach the tube to a telescopic antenna pole. I then laid out some wire radials, partly because it was easier, and partly because then I could more directly compare a wire vertical to a custard one.

The Custard Feedpoint – the green wire enters only as short distance into the tube

The next step was to attach an antenna analyser and see how well the monstrosity my creation functioned as an antenna. To my surprise, it was not resonant in any frequency below about 150mhz, but above that, was resonant on many frequencies. The frequency response was described by one person on the London Hackspace Radio IRC channel as “gloopy”

Unfortunately, I didn’t have any equipment with me for the 2m/145MHz band (actually I did, but didn’t need have the necessary coax adaptors), so I swapped to a more regular wire radiating element and made some 20m HF contacts (mostly FT8, a couple of SSB voice ones as well) to avoid the trip out to the nearby field.

Custard’s Last Stand

The following day, I returned with much the same setup, but this time also with a Yaesu VX-7 handheld and after putting up the pole again and laying out some (longer) radials, I tried to open a local repeater, GB3EL – and it worked! A few test calls later, and I realised I was able to consistently open the repeater and get a signal of about S6-7 back from it on the custard antenna. Just to test, I disconnected the custard and got nothing- so the radials and short stub of wire at the feedpoint were not enough on their own – the custard was clearly radiating!

After this, I switched back to a normal antenna – I had proven the concept and wanted to use the limited daylight left for more usual contacts. I didn’t feel like wasting too much time on what I suspected to be an inefficient 2m antenna in an area where 2m isn’t always very effective.

Analysis and Conclusion

It took me a while to figure out what was actually going on here – the custard seemed to be resonant on a much higher frequency than its length would suggest – either the speed of light is faster in custard – opening up a whole new branch of custard physics, or something else is going on. As cool as new custard physics would be, I think there’s a more rational explanation.

A brief detour into antenna theory

Before I carry on, a brief word about impedance of antennas. Impedance is the combination of resistance and reactance. Resistance affects all frequencies equally (including DC) whereas reactance is frequency-dependant. Reactance also introduces phase shifts. Reactance is caused by capacitance and inductance. We can ignore the reactance here because while it is important to the overall matching of the antenna, it does not play much of a role in the actual losses in the system. In other words, when I talk about resistance here, I really do mean resistance – I’m not misnaming impedance!

A simple model of the losses in an antenna

In order to efficiently radiate, an antenna needs to be matched to the transmitter. (I had written a whole section here explaining why, but it’s actually irrelevant for this discussion – maybe that can become a future post!) Transmitters are typically designed to put power into a 50 ohm load. That 50 ohms could be a resistor (in the case of a dummy load) or it could be a perfectly matched antenna. In reality most of the time you have some combination of this. In fact, you can model the antenna system as two resistors in series – one representing the electrical resistance (losses in the wires and connectors) and the other representing the radiation resistance (losses due to RF emissions – this is the bit we want!). The trick for an efficient antenna is to minimise the electrical resistance and maximise the radiation resistance. As an example, if you have an antenna with a loading coil, the coil contributes to the electrical resistance, but does not contribute much to the radiation resistance. I should mention this concept made a lot more sense to me after reading the book “SOTA Explained” by Jamie Davies – Here’s a link on Amazon, though it’s also available from the RSGB store.

Back to the main topic

I suspect what’s actually going on here is that the custard is quite lossy – so the RF only actually “sees” the bottom part of the antenna – in effect, it behaves like an antenna made of high resistance wire – the lower part provides a combination of electrical and radiation resistance, but as you travel up the column of antenna fluid custard, the electrical resistance takes over until eventually there is basically no radiation. Modelling it in the same way gives us something like this:

Model of “losses” in a custard antenna system

Practically, this means that while the custard works as an antenna, it cannot make antennas for lower frequencies since the losses are too great, and the top part of the custard simply doesn’t take part in the radiation.

In summary, Yes, you can use custard as an antenna, no I wouldn’t recommend it unless it’s all you have. If it is all you have though, and you need to get a signal out on about 145MHz, sure, go for it!

Custard Antenna

Some time ago, someone said of their ex-military antenna tuner “it would tune a bowl of custard if you asked it to” – this got me thinking, could I make my MFJ-971 antenna tuner literally tune a bowl of custard?

I’ve always been interested in strange antennas, having stumbled upon the K0S Strange Antenna Challenge many years ago. Sadly, the original website no longer seems to be around, however there are still references to it online, such as a post from the organiser, Erik Weaver N0EW, on eham: Essentially, it’s a special event station that uses only non-standard items as antennas. The example given was that of a tent pole: if you just stick the tent pole in the air, that’s not really a strange antenna since it’s just a bit of metal. If, however, you put the whole tent up in a tree, that’s a strange antenna.

For quite a long time, I’ve been meaning to try out some unconventional antenna ideas, and this seemed like the perfect chance! It’s more than just a joke though, it’s also a good opportunity to learn something about antennas.

Early Experiments

I decided to start small, so I first conducted a small experiment with a pot of custard. I actually bought two: one to taste and the other to shove some wires in and try to tune. Good job as well – one of them turned out to be green and furry when I opened it, so I ended up eating one spoonful of the correctly coloured one, before proceeding to tune it.

It worked! I managed to tune a pot of custard to the 20m band.

Full Size Bowl

I decided that to satisfy the spirit of “tuning a bowl of custard” I needed a literal bowl of custard. I also wanted a relatively large amount of custard to give it the best chance of doing something interesting. I went to the local supermarket and bought a 4 litre bowl and a few 1kg boxes of custard.

Three boxes of antenna fluid (aka custard)

This sat around for a couple of weeks, generally getting in the way until I finally got around to trying to tune a bowl of custard. The other problem was I couldn’t quite believe how daft the thing I was about to do was.

Finely tuned dessert

Anyway, one Saturday evening, I finally got chance to fill a bowl with custard and have a go. In the end I managed to get three boxes of custard into the bowl (I’d actually bought more, so I have some spare – more on that later) and tuned it up on 20m (14.15MHz to be specific – though it was flat across the band). Just out of interest, I hooked up my Yaesu FT-817ND and tried to receive some FT8. In case you haven’t come across it, FT8 is a digital mode designed for very weak signal reception. Some people hate it, but I really like how well it demonstrates band conditions, including how they change very quickly. Anyway, I figured if it was going to work on anything, it was FT8. To my surprise, after enabling the receiver’s preamp and turning up the RF gain, I was able to receive a signal from Greece – at one point the signal was -6dB which for FT8 is very much a decent signal! It’s worth mentioning here that on receive the ATU really doesn’t do much. But nonetheless, I was receiving FT8 on a bowl of custard! I did swap the custard out for a dummy load, but unsurprisingly, it did not receive anything – so at least on receive, the custard was not a complete dummy load. I do wonder if some of the pickup was on the wire feeding it, but the custard was definitely doing something.

FT8 reception on custard

In any case, the question was not if you could use custard to receive, but rather if it was possible to tune a bowl of custard to an acceptable SWR.

I did attempt to put about 30-40W of RF into the antenna system (bowl of custard + ATU) however nothing was picked up on PSKReporter, leading me to conclude that the bowl of custard was significantly inferior on transmit compared to receive. This came as absolutely no surprise of course, but I had to try.

What Next?

There are a couple of things I’d like to try next. The first is to repeat the experiment with a bowl of jelly, to see if it behaves any differently.

The second experiment I would like to try, and the subject of the rest of this post, is to make a custard antenna that can at least vaguely usefully be used as an antenna. I want to make a contact of some form on an antenna made of custard!

My initial idea was to take some plastic guttering and make a custard dipole. The problem here though is supporting the guttering far enough above the ground for it to be usable as an antenna. My next thought was to use rigid plastic pipe, filled with custard, however I’m a little concerned that once filled with custard, the pipe could become too heavy to sensibly hold up. Since the length is probably more important than the diameter, I came up with the idea of using clear plastic tube – partly because it would look cooler if you could see the custard. Since custard is quite a thick material, I suspect I would need to use, maybe 15-16mm inside diameter tube to be reasonably sure of being able to fill it – and clear tube would be an advantage here since you would be able to see any air bubbles and squeeze them out.

I think aiming for an antenna resonant on the 20m band is perhaps the best bet. I could maybe make something for VHF but that’s not as fun, and based on a very quick test I did, custard might actually be lossier above 30MHz or so. Higher HF bands aren’t really very usable at this low point in the solar cycle, and I think 5m or so of custard is probably just about achievable.

There are a few questions remaining about this:

  1. Should I use custard for the radials as well?
    • I could maybe fill some tube (perhaps thinner tube) with custard as radials.
    • Or maybe I could use jelly radials?
    • Or maybe just wire? Is that cheating?
  2. What is the velocity factor of custard? Not a question I ever thought I’d have, but one I should be able to approximately answer if I build this thing.
  3. How much would a 5m tube of custard weigh?
    • Could a fibreglass pole support it or does it need some thing stronger?
    • How much antenna fluid custard do I need to fill the tube?
  4. How lossy is custard as an antenna?

The answers to these questions (which no one has ever asked) will come in a future post.

Raspberry Pi Clustering – Part 1

Note: The Pi3 figures in this may be questionable since I have since discovered it was overheating and scaling its clock. A future post will address this. Since I’m interested in real-world performance though, I’m not going to completely discount these results since overheating is a factor to consider!

The other day I came up with a slightly daft idea: deploy the website I’m currently developing to a cluster of raspberry pis rather than my usual Linode setup. Seems like a crazy idea? Yeah, probably! But I still want to try it!


My linode setup is getting a bit complicated. Right now I have various mostly unmaintained sites running on a selection of docker containers, but this has its own problems: notably running out of disk space on the fairly small root partition. This whole setup needs rebuilding really. I’m tempted to just deploy it all with Ansible directly to the host. What does this have to do with pi clustering? well a pi cluster is a nice way of trying this out!

There Is Some History Here!

The Raspberry Pi website has itself been hosted on pis a couple of times, most recently it was announced the Pi3 launch was partly hosted on Pi2s!

It’s also possible to get hosting provided on a Pi3 but part of the fun of this idea is clustering! (though I will do some tests to see how it compares to a single pi3)

The original pi was said to not be a good choice since its performance for cost wasn’t good enough. That said, the zeros I’m using here aren’t actually too far off!

The setup: My Initial Idea

My first attempt was to deploy onto a ClusterHat ( with some zeros, an old model B+ (512Mb ram version) and a Pi3 for good measure and comparisons. The setup was something like this:

Diagram showing the connection between the pis

My initial setup

What do the parts do?

Component Purpose
HAProxy Load balancer. Distributes the load among the other servers.

Nginx can also do this, but I chose HAProxy because it supports easily disabling nodes for maintainance (see this ansible example)

Nginx Web cache. Reduces load on the web app by serving pregenerated responses where possible
WebApp This is the Spark Java (not Apache Spark!) based app I want to deploy. Not the best choice maybe, but I decided to try it before deciding on using a pi!

The thinking behind this setup was that the webapp would be slow (it is!) but the nginx server on each node can be a cache. haproxy then load balances between these caches. When stress testing with JMeter it became clear that the HAProxy install on the head node was actually the bottleneck here. Still let’s see what this is capable of!

I used Apache Benchmark to get some ideas of the performance of this :

$ ab -kc 100 -n 10000 http://piclusternode0/
This is ApacheBench, Version 2.3 <$Revision: 1796539 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation,

Benchmarking piclusternode0 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: nginx/1.10.3
Server Hostname: piclusternode0
Server Port: 80

Document Path: /
Document Length: 556 bytes

Concurrency Level: 100
Time taken for tests: 11.841 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 0
Total transferred: 9144496 bytes
HTML transferred: 5560000 bytes
Requests per second: 844.50 [#/sec] (mean)
Time per request: 118.413 [ms] (mean)
Time per request: 1.184 [ms] (mean, across all concurrent requests)
Transfer rate: 754.16 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 2 43 203.7 21 3165
Processing: 6 75 49.2 67 988
Waiting: 6 75 48.6 66 983
Total: 12 118 207.8 88 3215

Percentage of the requests served within a certain time (ms)
 50% 88
 66% 91
 75% 93
 80% 94
 90% 108
 95% 173
 98% 433
 99% 1152
 100% 3215 (longest request)

Thoughts On First Results

That’s not actually bad for a cluster of low power machines. Running it a couple of times and watching the CPU load on the load balancer suggests the bottleneck here may again be the HAProxy node.

It turns out that HAProxy is mostly single threaded. It is possible to make it multiprocess, but then you lose some of the ability to control it and get metrics out of it. I adjusted the config to run four processes (the pi2 has four CPU cores) but the results were actually quite similar so I guess it was just at its single core limit or maybe the network bandwidth was becoming a problem.

In reality, this isn’t a very realistic situation for two reasons though:

  1. Not all hits will be to the homepage – some will require loading data via the webapp and so will not be cachable
  2. I intend to use TLS (aka HTTPS) for this site eventually, which will significantly increase load.
  3. The site I want to host on this will involve some POST requests, requiring database writes. This means the load will be a mix of IO and CPU-bound requests.

How Does This Compare To A Single Pi?

It’s worth stopping to consider how one Pi performs. Here are some numbers:
(All 10000 connections, 100 at a time) Note these kept changing as I tested, so they’re not really incredibly accurate.

Metric Cluster Single Pi Zero Single Pi 3
Mean Time Per Request (ms) 118.413 225.442 62.771
95% Time (ms) 173 280 62
Requests Per Second (mean) 844.50 443.57 1593.10

So… A single Pi3 does a better job – that’s actually not all that surprising given it’s more powerful than the average node. I tried increasing its weight in the cluster, but I think it is hitting IO bandwidth limits on the balancer node.

Perhaps a better test would be to hit an endpoint that does not benefit from caching. This is going to stress the CPU far more.

Metric Cluster Cluster (No Nginx) Single Pi Zero Single Pi Zero (No Nginx) Single Pi 3 Single Pi 3 (No Nginx)
Mean Time Per Request (ms) 1058.234  430.534 6354.980  6588.006 1877.685 236.815
95% Time (ms) 5476  1187 7060  7384 5053 655
Requests Per Second (mean) 94.50  232.27 15.74  15.18 53.26 422.27

It’s worth noting the pi3 wasn’t actually that heavily loaded (about 50% per core) – It seems the nginx proxy (which in this case was just passing data through) might be the limit here hence I did the same test but going directly to the Java webapp. I suspect the config I wrote to optimise caching by blocking multiple requests until one origin request has completed is limiting performance more than I expected! On the zero this has no effect though, because it’s single-cored.


It seems clear that you’d need a very large number of zeros to beat a Pi3 – but I kinda knew that anyway. It does seem though like for CPU-bound workloads a cluster of Pi3s (or similar – more on that later) could be an advantage.

It seems for HTTP easily cached workloads, the cluster is really no benefit since the IO bandwidth of the load balancer pi is lower than that same pi would host directly. For more CPU-bound workloads though, it could help.  I’m going to keep working on this idea, including:

  • Using Orange Pis (if I can get them to work that is!)
  • Using a Pine64 as the load balancer
  • Putting Nginx the other side of HAproxy
    • The theory being for anything the cache is useful for, a single pi can cope
    • As I mentioned before, HAProxy is a bit nicer to configure for updates
    • I suspect this won’t help once TLS is added though
  • Combinations of these
  • Add TLS into the mix (since every site should these days!)

Reasons Why I Shoot Film

This post is the first in a hopefully regular series of posts about film photography.

Inspired by a combination of Boredom, YouTube videos, and that nagging feeling I should do something with those rolls of film I put through a Holga a couple of years ago, I’ve been shooting a fair bit of film recently. In fact, I have acquired more than one film camera from eBay recently. Mostly inexpensive, and most recently one not so inexpensive one.

Continue reading