Sean Elvidge

Space Weather - The Musical

2026-06-17T22:00:00+00:00

What does a geomagnetic storm sound like?

Not metaphorically. Literally.

Imagine taking the indices we use to drive our models of near-Earth space (solar radio flux, sunspot number, geomagnetic activity, ring-current disturbance) and translating it into a piece of music. This isn’t a “space-inspired” soundtrack, but a deterministic piano score where every pitch, rhythm, chord, tempo change and accent is driven by real space weather indices.

That is what this blog post is all about. About how we turn space weather events into piano music. The result is a structured musical translation of the space environment, built so that the data can be heard.

The May 2024 Superstorm

(Click play and listen along whilst you read the rest of the post explaining where the music comes from)

A storm, compressed into the hands of a pianist

The example above uses data from May 5th to 15th 2024. This 10-day window captures the period around the major May 2024 geomagnetic storm — an event of intense aurora, severe geomagnetic activity and unusually (at least in recent times) dynamic near-Earth conditions.

Musically, the piece is placed onto a fixed daily grid:

one day becomes four bars;
each bar is in 6/4;
one day is therefore 24 quarter-note beats;
one hour corresponds to one quarter-note beat;
thirty minutes corresponds to half a beat.

So however expressive the notes become (more details below), the structure remains anchored in time. Every layer of space weather indices realigns at midnight. Each day takes up the same “musical space”, allowing the changing behaviour of the Sun-Earth system to become clear through changes in texture, pitch, rhythm and intensity.

Five indices, five jobs

Here we use five space weather indices, and each are given a different job to do in defining the music:

F10.7: the adjusted solar radio flux, controls the harmonic root in the left hand.
Sunspot number: adds ‘density’ to the left-hand. When the sunspot number is sufficiently high, we add an octave root.
Hp30: drives the right-hand melody. Because Hp30 is available every 30 minutes, it provides a natural melodic line.
Kp: controls the broad rhythmic regime. Quiet geomagnetic periods lead to slower figures; storm periods create denser patterns.
Dst: as it becomes more negative, the music becomes more forceful through dynamics, accents and tempo.

I was keen to use as many indices as possible to give the music as much ‘texture’ as I could do - entirely deterministically. I have tried to make solar activity shape harmony, geomagnetic activity shape motion and storm intensity shape tension.

The right hand melody: Hp30

The right hand is the most active. It uses a pitch drawn from F# minor (my favourite), ranging from F#4 to E6. Each Hp30 value is normalised and mapped to one of fourteen pitches.

This means that higher geomagnetic activity tends to push the melody upward. But the mapping is not absolute. To create the score we do some local normalisation, i.e. we care both about how large Hp30 is on a physical scale and how large it is relative to the other values in the selected date range. This was needed so that even quite periods (geomagnetically) stay interesting.

The Kp (a logarithmic scale from 0 to 9) is used to define the rhythm:

below Kp 5, the rhythm is relatively slow;
from Kp 5 to 7, the rhythm becomes more active;
above Kp 7, each hour becomes a four-note semiquaver figure.

But the exact rhythm inside each hour also responds to local movement in Hp30 and Dst. If Hp30 jumps within the hour, or if Dst changes sharply from one hour to the next, the right hand becomes more animated. A quiet Kp period is therefore not forced to be dull if the other data are still moving.

The left hand: harmony and tension

The left hand provides the harmonic ‘frame’ of the piece. It chooses roots from an ordered set:

F#, A, B, C#, D, E, G#.

These are not arranged chromatically but arranged to give useful harmonies inside the F# minor world we’re working in.

Each day has four bars, and each bar receives one harmonic root. F10.7 selects the base root. Then the day’s pattern of F10.7 and sunspot-number change determines how the four roots move. If solar activity and sunspots are both rising, the harmony tends to climb. If both are falling, it descends. If they disagree, the progression takes a mixed path.

The chords themselves are simple diatonic triads: F# minor, A major, B minor, C# minor, D major, E major and G# diminished. This keeps the harmonic language coherent while still allowing the data to move the music through different regions of the scale.

Dst and Kp then decide how much weight the left hand carries. In low-tension periods, a bar may simply be one long chord. In moderate tension, a bass note is added before the chord. In high tension, the left hand pulses with repeated bass-plus-chord figures. A storm is therefore not only heard in the treble melody but also changes the ‘weight’ of the piano texture.

Tempo, dynamics and accents

The score also has some performance notes in it. Whilst the overall tempo of the piece remains constant throughout, average note length is (slightly) determined by solar activity, Hp30, Dst, and Kp can all push the tempo upward (having an impact of making the piece feel like the speed is changing, but only really ranges from about 60 to 120 bpm so remains playable (at least to my limits!).

We also use four dynamic levels: p, mp, mf and f. This allows for clear changes in intensity without going over the top. Accents appear when the normalised activity is high enough. Staccato is added to the shortest right-hand notes. During intense storm intervals, the music becomes not just higher or faster, but sharper and more articulated.

Why this is more than sonification

Many data-to-music projects work by assigning one variable to pitch and another to volume. That can be effective, but it often produces a thin musical result. This piano mapping is a little more ambitious. It treats the space weather system as a set of interacting musical lines:

long-timescale solar conditions become harmony;
short-timescale geomagnetic changes become melody;
storm thresholds become rhythmic regimes;
Dst depression becomes tension;
sunspot number becomes chord density;
daily evolution becomes harmonic progression.

This is also all deterministic. Given the same date range then the same notes, rhythms, dynamics and accents are produced. There are no random choices. This is important because it means the musical output is reproducible. The piece can be discussed, analysed and regenerated.

Hearing the May 2024 storm

In calm moments of the space environment, the music has ‘space’. The left hand can hold long chords while the right hand moves slowly through the scale. As activity increases, the melody becomes more agile. Rhythms subdivide. The tempo rises. Accents appear. The left hand gains pulse. The storm becomes audible not as noise, but as structure.

That is the most compelling aspect of the approach. Space weather is often communicated through plots, maps, alerts and indices. Those are essential. But music offers a different route into the same system. It lets us hear change, pressure, release, escalation and recovery.

The May 2024 storm was a scientific event, an operational challenge and, for many people, a spectacular visual experience. In this piano version, it becomes something else as well: a short, reproducible musical portrait of a disturbed geospace environment.

Who wins in a 48 team World Cup - Panini

2026-06-11T19:30:00+00:00

Since 1970 in Mexico Panini have published the official World Cup sticker book. It has become synonymous with the tournament, with a huge following and has become a multi-billion dollar business. It is estimated that for this World Cup between 500 and 800 million sticker packs will be sold [1].

So what does it take to complete the sticker book? And how has this changed from the last 32-team World Cup (Qatar 2022) to this, the first 48-team World Cup? Well, the 50% increase in teams, “only” leads to a 10% increase in required stickers (but a 42% increase in costs…).

For this analysis we assume that stickers are randomly distributed in packs and are each equally likely (which has been claimed to be the case by the Panini CEO, however research has shown that ‘shiny’ stickers are systematically rarer than others by about 2x).

To see where these numbers come from, let \(N\) by the total number of stickers in the album and \(k\) be the number of stickers in one pack. So each pack contains \(k\) distinct stickers which are sampled uniformly at random from the \(N\) possible stickers. So each pack is equally likely to be any one of

\[{N\choose k}\]

which means “\(N\) choose \(k\)” and, simply, is used to represent the number of ways to choose \(k\) items from a set of \(N\) items where order does not matter. You can calculate its numerical value using:

\[{N\choose k} = \frac{N!}{(N-k)!k!}\]

where “!” means factorial.

So the number we are trying to find is:

\[E_m = \text{ expected number of additional packs needed when you already have } m \text{ distinct stickers}\]

and in our specific case the goal is to compute \(E_0\) because we start with zero (0) stickers. The boundary condition is \(E_N = 0\) because once you already have all \(N\) stickers, you need to more packs. Now suppose you currently have \(m\) distinct stickers, then

\[N-m\]

stickers are still missing. In the next pack of \(k\) stickers, suppose exactly \(r\) of them are new stickers. To get exactly \(r\) new stickers:

choose \(r\) stickers from the \(N-m\) missing stickers,
choose the remaining \(k-r\) stickers from the \(m\) stickers you already have.

So the number of packs that contain exactly \(r\) new stickers is:

\[\binom{N-m}{r}\binom{m}{k-r}.\]

Since all possible packs are equally likely, the probability of getting exactly \(r\) new stickers is:

\[p_{m,r} = \frac{\binom{N-m}{r}\binom{m}{k-r}}{\binom{N}{k}},\]

where invalid binomial terms are treated as zero. Then, the possible values of \(r\) are:

\[\max(0,k-m) \le r \le \min(k,N-m)\]

because you cannot get more new stickers than are missing, and you cannot get more than \(k\) new stickers from one pack. So now we need fewer stickers to complete our collection, specifically we move from ‘state’ \(m\) to state \(m+r\) with probability \(p_{m,r}\). Therefore:

\[E_m = 1 + \sum p_{m,r}E_{m+r},\]

where the “1” represents the pack you just bought. However, one possible outcome is \(r=0\), meaning the pack we bought contains no new stickers. In that case we remain at state \(m\), so \(E_m\) appears on both sides:

\[E_m = 1 + p_{m,0}E_m+\sum_{r\ge 1}p_{m,r}E_{m+r},\]

which we can rearrange to get:

\[\begin{eqnarray*} E_m - p_{m,0}E_m &=& 1+\sum_{r\ge 1}p_{m,r}E_{m+r},\\ E_m(1-p_{m,0}) &=& 1 + \sum_{r\ge 1}p_{m,r}E_{m+r},\\ E_m &=& \frac{1+\sum_{r\ge 1}p_{m,r}E_{m+r}}{1-p_{m,0}}. \end{eqnarray*}\]

This gives an exact recursive calculation. Since \(E_m\) depends only on \(E_{m+1}, E_{m+2},\ldots,E_N\) we compute backwards from:

\[E_N=0\]

down to

\[E_0.\]

In 2022 each Panini pack contained 5 stickers and the whole album required 670 stickers. So that gives \(N=670\) and \(k=5\), using the recurrence relation described above gives:

\[E\approx 946.9837488\]

so the expected number of packs required to complete the album is 947 packs.

In 2026, each pack contains 7 stickers and an album needs 980 stickers. So with \(N=980\) and \(k=7\), and using the defined relation the expected number of packs is:

\[E_0 \approx 1042.36\]

or about 1042 packs. The increase from 947 packs to 1042 packs is an increase of about 10% (10.031…% to be more precise).

A multi-pack of stickers at my local supermarket costs £7.50 for 6 packs. To get the expected number of 1042 packs requires 174 multi-packs at a cost of £1,305.

For comparison, in 2022, a 6 pack multi-pack (remembering each contained 5 stickers per pack), cost £4.99, so to get the expected number of 947 packs, 158 packs were needed, at a total cost of £788.42. Adjusting for inflation that comes to £918.89.

So whilst we only need 10% more stickers, we should expect to spend 42% more to complete the album.

When Good Ideas Don’t Survive Contact With Data - A December Football Myth Tested

2025-12-31T19:30:00+00:00

Every so often I come up with what feels like a great idea for a blog post. A clever angle, an interesting hypothesis, something that surely must be hiding in the data just waiting to be revealed. And then, after hours of analysis, the result is… nothing. Completely flat. No effect.

This is one of those stories.

I wanted to look at whether the famously congested English football December schedule gives an advantage to teams (either weaker or stronger teams). With matches every few days, tired legs, rotation, winter weather, and general chaos, it seemed plausible that perhaps squad rotation from the stronger sides might be key, or perhaps stronger sides might stumble more than usual and that underdogs might pick up a few unexpected points.

It’s a nice idea. Unfortunately, as we’ll see, a nice idea does not guarantee a nice result.

Why December Might Matter

Consider what happens in December in English football:

Fixture congestion increases dramatically.
Elite teams face multiple competitions and may rotate heavily.
Injuries accumulate and squad depth becomes critical.
Weather worsens, pitches soften, and match conditions become less predictable.

Perhaps if stronger teams depend more on structure and precision, and weaker teams rely more on disruption and variance, you can build a reasonable argument that December’s chaos environment might narrow the gap.

In other words, if \(S\) is team strength and \(\epsilon\) is “football randomness”, maybe the effective strength looks more like:

\[S_{\text{effective}} = S + \epsilon,\]

and perhaps December has a larger \(\epsilon\).

If so, underdogs might perform better than expected.

How to Measure “Underdog Performance”?

To test this we need a very large dataset of football results and a way to quantify which team was stronger before the match was played.

In the team strength database a higher values means a stronger team.

Thus:

The stronger team is whichever side has the higher rank.
The weaker team (the “underdog”) is the side with the lower rank.

And for each match, we compute:

Underdog points
Underdog goal difference
Whether the match took place in December

Everything you’d need to test whether December helps the weaker (or stronger) team.

December vs Non-December: The Raw Numbers

Aggregating across every division, every season, and every match where a clear favourite exists, we get:

Outside December, underdogs earn: 1.178 points per match
In December, underdogs earn: 1.181 points per match

The difference, 0.0026 points, is essentially zero.

The underlying distribution of goal differences tells the same story.

I will admit, this was mildly disappointing. But maybe aggregate data hides the truth? Perhaps the Premier League behaves differently? Or lower leagues, with smaller squads, show something?

Nope.

However you slice the dataset, Tier 1, Tier 2, Tier 3, Tier 4, the same conclusion emerges:

The difference between underdog performance in December vs other months is tiny.
In every tier, the effect is statistically insignificant.
Some tiers lean slightly positive, others slightly negative, all within noise.

So weaker teams, as a group, do not seem to benefit from the December schedule.

You might reasonably ask whether the favourites show any seasonal change. If weaker teams don’t improve, perhaps stronger teams deteriorate?

I ran the same comparison for the stronger side in each match.

The result? Exactly the same story.

Stronger teams’ points and goal differences in December are statistically indistinguishable from the rest of the season. No noticeable dip, no seasonal weakness, no hidden December curse.

In short:

Neither underdogs nor favourites show any meaningful change in performance during December.

A More Formal Statistical Model

To be thorough, I fitted a regression model of underdog points:

\[\text{pts} = \beta_0 + \beta_1\,\text{December} + \beta_2\,\text{Home} + C(\text{Tier}) + C(\text{Season}) + \varepsilon .\]

This controls for:

Home advantage
League tier
Season-by-season variation

The coefficient for the December effect came out as:

\[\beta_1 = -0.0018 \pm 0.013,\]

which is, again, indistinguishable from zero.

When a Good Hypothesis Fails

This leads to what I think is the most important lesson from the whole exercise.

In science many ideas do not survive contact with real data. They’re plausible, they’re elegant, and they would make for a great story, but the universe simply refuses to cooperate.

That does not make the work wasted.

It makes it honest.

The Academic Problem With Negative Results

In academia, “negative” or null results are notoriously hard to publish. Journals often prefer dramatic or surprising findings, which unintentionally encourages selective reporting:

Effects that don’t exist are quietly forgotten.
Effects that appear by chance get published.
The literature accumulates exciting stories but not necessarily accurate ones.

This December analysis is a textbook example: an interesting idea, diligently tested, clearly unsupported.

Yet these non-effects matter. They give us a clearer picture of reality, and they remind us that the absence of a pattern is itself information, and sometimes, important information.

Christmas Day Football - The Lost Tradition

2025-12-25T09:00:00+00:00

It’s been exactly 60 years since the last English league match was played on Christmas Day. On 25 December 1965, Blackpool hosted Blackburn Rovers in a festive Lancashire derby that would become the final flicker of a long-standing tradition. While Boxing Day remains a staple of the football calendar, the idea of heading to the ground after unwrapping presents has quietly faded into the past. Here’s a look back at how Christmas Day football began, what made it special, and why it came to an end.

Victorian Beginnings

The Football League didn’t include Christmas Day matches in its inaugural season (1888), but by the very next year, Preston North End hosted Aston Villa on 25 December 1889. That 3-2 win for Preston drew around 9,000 fans and marked the beginning of Christmas football as a regular fixture.

By the early 20th century, the tradition was well-established. Local derbies were often scheduled to reduce travel, and many clubs played the same opponent home and away over Christmas and Boxing Day. It wasn’t unusual for fans to watch a game in the morning and be home in time for turkey in the afternoon.

When Christmas Meant Football

For much of the 20th century, football on Christmas Day was as normal as carols and crackers. The interwar years and post-WWII period were the high points, with full fixture lists and packed crowds. In 1949, over 3 million fans attended league matches during the Christmas week. Players would sometimes play three matches in four days. Fans shared flasks and sang carols on the terraces.

Holiday scheduling quirks added to the charm. Tranmere once lost 4–1 on Christmas Day, only to win the return fixture 13–4 the next day. Matches were sometimes played in snow, fog, or biting cold, but still the crowds came.

Christmas Day 1914 was particularly notable. Despite the shadow of World War I, nine First Division matches went ahead, drawing a combined crowd of around 173,000. Football was still seen as a morale-booster, and a newspaper at the time declared that to millions:

Christmas without football would not be Christmas at all. (Essex Newsman, 21 December, 1945)

And across the Channel that same day, one of football’s most poignant and enduring moments occurred. British and German troops along the Western Front took part in the famous 1914 Christmas Truce. In the frozen fields of Flanders, soldiers from both sides briefly set down their arms and came together to sing carols, exchange gifts, and even play informal matches in No Man’s Land.

Christmas Day Classics

There were high-scoring thrillers (Chelsea 7–4 Portsmouth in 1957), and even a match in 1940 where Norwich beat a scratch Brighton team 18–0. The largest Christmas Day crowds topped 60,000 at places like Newcastle and Arsenal.

Some stories veer into folklore: players turning up tipsy after too much Christmas cheer, supporters pelting the pitch with orange peel, and squads sharing a turkey dinner on the train between back-to-back games.

And for Coventry fans: our own Ken Satchwell scored the last ever Football League hat-trick on Christmas Day, in a 5–3 win over Wrexham in 1959. One of the last festive hurrahs before the tradition faded.

The Final Whistle: 25 December 1965

60 years ago today, in 1965, Blackpool beat Blackburn Rovers 4–2 in front of just over 20,000 fans. Young Alan Ball, not yet a World Cup winner, scored one of the goals. The weather was chilly but the game was lively, and the matchday programme was wrapped in festive green and tangerine.

That game became, quietly, the last of its kind.

Why It Ended

The reasons are a mix of culture, logistics, and changing times. More families preferred to stay home on Christmas morning. Public transport no longer ran on Christmas Day. Policing and stewarding became harder to organise. Players were increasingly reluctant to spend Christmas away from their families.

The spread of floodlights and TV also meant fixtures could be moved to evenings or rescheduled around the holidays. Boxing Day, a public holiday since the 19th century, proved more popular for fans and clubs alike. By the early 60s, most clubs had already opted out. No ban was needed. The tradition just slipped away.

Remembering Christmas Football

Christmas Day football was sometimes chaotic, often charming, and always memorable. It belongs to another era, one of packed terraces, handwritten programmes, and roaring fires waiting at home. Brentford tried to make it come back in 1983 when the tried to schedule a game for Christmas morning. But fans rebelled and the club backed down.

These days, players can enjoy their mince pies in peace, and fans save their voices for Boxing Day.

But 60 years ago today, Blackpool beat Blackburn in the last of 1,243 Christmas Day fixtures, as English football took its final bow on December 25th.

How to Calculate League Position Probabilities

2025-12-22T19:30:00+00:00

By mid-season, football fans instinctively start doing probability in their heads.

If we win our next two, and they drop points away at… actually, hang on.

This usually ends with a napkin full of scribbles and a strong emotional commitment to a particular set of results. What follows in this post an attempt to do the same thing — but with statistics.

On the predicted league tables page of this site you’ll find the probabilities for every team finishing in every possible league position. Not via vast simulations. Not using “expected points” (xPts). Not vibes. Actual probabilities.

This post explains how those tables are built, what assumptions sit underneath them, and, importantly, what they don’t claim to do.

Estimating the future

For each remaining fixture, we estimate the probability of:

a home win,
a draw,
an away win.

These probabilities come from my rating model that combines:

team strength,
home advantage,
historical era-dependent draw rates,

(you can try the match probability calculator yourself).

For these forecasts we assume that team strength remains constant for the rest of the season. This is not because teams don’t change, obviously they do, but because this assumption lets us ask a very precise question:

“Given what we know right now, what does the future look like?”

No form, no momentum, no injuries, no managerial bounce. Just the present frozen in amber.

Turning matches into points

Once we know the probability that a team finishes on, say, 64 points, we can ask the next question:

“What position will that correspond to?”

For each possible final points total:

we calculate the probability that other teams finish above that total,
the probability they finish below it,
and the probability they finish on exactly the same points.

When teams are tied on points, we assume a random tie-break between them. This is not how the real leagues work, goal difference matters, but it is a deliberately neutral assumption that avoids injecting additional modelling choices.

By aggregating over all possible point totals, we obtain the probability that a given team finishes in position 1, 2, 3, …, N.

That’s what fills the table.

For example:

Why this isn’t a simulation

Many league predictors simulate the rest of the season thousands or millions of times (e.g. Opta). Those approaches can be flexible and intuitive. However this approach is different.

Here every probability is computed exactly so the results are fully reproducible and even small probabilities (like a 0.3% chance of winning the league) are real, not artefacts of random noise. However the cost of this precision is a stronger set of assumptions, here we assume:

match outcomes are independent,
team strengths do not change,
tie-breaks are random.

Reading the table

Each row shows a team; each column shows a finishing position.

A cell marked “12%” means exactly that: a 12% chance of finishing in that position.
“<1%” means possible, but very unlikely.
“–” means impossible — the team cannot mathematically finish there.

The strongest probability in each row is highlighted, and other positions are shaded relative to it. This gives a quick visual sense of where a team’s likely finishing range lies.

This can be hard to see on a mobile scream, so the full table is replaced by a compact summary:

chance of finishing 1st,
chance of finishing Top 6,
chance of finishing in the Bottom 3.

The full table is always available via a toggle, and the entire table can be downloaded as an image for closer inspection.

What this doesn’t say

These tables do not say:

who will win the league,
that form and injuries don’t matter.

They say something subtler, and, I think, more interesting:

“If the football world stays roughly as it is today, how wide is the space of possible futures?”

Sometimes that space is narrow. Sometimes it can be surprisingly large.

Check out the predicted league tables for yourself. Updated everytime the football database is updated.

Tracking Football Team Strengths with a Bayesian Kalman Model

2025-12-15T09:20:00+00:00

Not all football-rating systems are the same. Many of the public ones, like the excellent ClubElo, do a fine job of ranking teams using elegant updates. Win and your rating rises, lose and it falls; the amount you move depends on how “surprised” the model was.

This model begins in the same spirit but replaces those heuristic updates with a probabilistic engine: a Bayesian Extended Kalman Filter coupled to a modern version of the Bradley–Terry model (with added draws). The core idea is simple: treat each team’s strength as something hidden that we estimate and track through time, with uncertainty that expands between matches and contracts when new results arrive.

Across the full historical dataset (every English league match since 1888) the system achieves a mean Brier score of 0.2035. These values are significantly better than many published computer models (e.g. Nyamdorj et al. 2014, BSIC, 2024 and Harvard Sports Analysis Collective, 2015) which indicates a stable, well-calibrated predictive performance. Crucially, because the filter quantifies its own uncertainty, it can tell us not only who is strongest, but how confident we should be in that judgement.

The rest of this post goes into the mathematical details of the ranking algorithm, but if you want to access the underlying data it is available here (specifically the file EnglandLeagueResults_wRanks.csv).

The Big Picture

Imagine every team has a hidden “true strength” \(s_i\). Before a match, the home and away teams carry beliefs about their current strengths, each with an associated uncertainty. When they play, the result provides new information that updates those beliefs.

In the ClubElo framework this update is written directly as

\[R_{\text{new}} = R_{\text{old}} + K(S - E),\]

where \(S\) is the score (1 for a win, 0.5 for a draw, 0 for a loss), \(E\) is the expected probability, and \(K\) is a fixed responsiveness parameter.

Here, the same logic is embedded in a Kalman filter, which means the effective \(K\) is learned automatically. The update size depends on two things: how uncertain we are about the teams, and how surprising the result was. Upsets between uncertain teams lead to large updates; shocks between well-understood teams barely move the needle.

Between matches, team strengths are not frozen. Instead, they evolve according to a mean-reverting stochastic process, allowing form to drift while preventing runaway behaviour. Newly promoted teams begin with large uncertainty and adapt quickly; established teams gravitate toward long-term baselines rather than rising indefinitely.

Modelling Match Outcomes (Wins, Draws, Losses)

Match outcomes are modelled using the Davidson extension of the Bradley–Terry model, which naturally incorporates draws. The probability of each outcome is

\[\Pr(\text{Home}) = \frac{e^{\Delta}}{e^{\Delta} + e^{-\Delta} + \kappa},\] \[\Pr(\text{Draw}) = \frac{\kappa}{e^{\Delta} + e^{-\Delta} + \kappa},\] \[\Pr(\text{Away}) = \frac{e^{-\Delta}}{e^{\Delta} + e^{-\Delta} + \kappa},\]

where

\[\Delta = \beta \bigl(s_H - s_A + h\bigr).\]

Here, \(s_H\) and \(s_A\) are the latent strengths of the home and away teams, \(\beta\) is a scaling parameter, \(\kappa\) controls the draw rate, and \(h\) is the home-advantage term.

Home Advantage - Explicit and Time-Varying

Home advantage is not treated as a fixed constant. Instead, it is explicitly modelled and allowed to vary over time. Historical analysis shows that home-win rates in English football have declined dramatically since the late 19th century, falling from well above 60% to closer to 40–45% in the modern era (see this analysis).

By allowing the home-advantage parameter \(h\) to evolve slowly with time, the model correctly distinguishes between a home match in 1890 and one in 2025. This prevents systematic bias when comparing teams across eras and is a key improvement over static-advantage rating systems.

Validation and Rating Scale

Predictive accuracy is measured using the Brier score, defined as the mean squared error between predicted probabilities and observed outcomes. Over the full dataset the score is 0.2035 (for the 2024/25 season it is 0.2085), indicating robust calibration both historically and in the present day.

Internally, the filter operates on a latent “skill” scale roughly spanning \(-3\) to \(+3\). However for presentation, these values are mapped linearly onto an Elo-style scale, centred on 1000 points, with elite teams reaching 1800+ and lower-league teams clustering about a thousand points below. This transformation is purely cosmetic; all inference happens on the latent scale.

Part II — If You Dare Read On: The Mathematics

1. State Evolution (Ornstein–Uhlenbeck Dynamics)

Each team’s latent strength evolves as

\[s_{i,t+1} = \rho\, s_{i,t} + (1 - \rho)\, \mu_{\text{tier}(i,t)} + \varepsilon_{i,t},\]

with

\[\varepsilon_{i,t} \sim \mathcal{N}(0, q_i).\]

Here, \(\rho\) controls persistence, \(\mu_{\text{tier}}\) is the long-term baseline for the team’s division tier, and \(q_i\) is the process variance. This formulation allows ratings to drift while remaining anchored to realistic league-level expectations.

2. Observation Model

Given predicted strengths \(s_H\) and \(s_A\), the model produces a probability vector

\[\mathbf{p} = \begin{bmatrix} p_H \\ p_D \\ p_A \end{bmatrix},\]

using the Davidson–Bradley–Terry equations above. The Jacobian matrix \(H\) is computed by differentiating \(\mathbf{p}\) with respect to the state vector \([s_H, s_A]^T\), enabling linearisation of the nonlinear observation model.

3. Extended Kalman Filter Update

Prediction step:

\[x^- = F x_t,\] \[P^- = F P_t F^\top + Q,\]

where \(F = \rho I\) and \(Q\) is the process-noise covariance.

Observation noise:

\[R = \operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.\]

Update step:

\[K = P^- H^\top \bigl(H P^- H^\top + R\bigr)^{-1},\] \[x_{t+1} = x^- + K (y - \mathbf{p}),\] \[P_{t+1} = (I - K H) P^-.\]

The Kalman gain \(K\) replaces the fixed \(K\)-factor of traditional Elo systems, adapting automatically to uncertainty and surprise. A secondary control loop monitors the normalised innovation squared to ensure statistical consistency over time.

Why This Matters

This framework enforces mathematical honesty. Uncertainty is explicit, calibration is measurable, and every parameter (\(\beta\), \(\rho\), \(q\), \(\kappa\), tier baselines, and historical home advantage) has a clear interpretation. Instead of a single magic constant, the model becomes a living system that adapts across seasons, divisions, and eras.

Football Bogey Grounds and How Statistics Can Prove Them

2025-12-12T19:04:00+00:00

Football supporters are never short of folklore. Some of it heroic, some of it tragic, and some of it mathematically suspicious. Among the more enduring tales is the bogey ground: that one venue where your club never quite manages to win, no matter how many times the fixture computer sends you there with fresh optimism, a new manager and a good run of form.

But one question always pops up in the back of my mind when I hear this: When is a bogey ground actually a bogey ground, and when is it just a trick of small numbers?

Most fans instinctively understand the point. Losing your only ever visit to Carlisle does not make Brunton Park a cursed ground. A couple of failed trips to Luton are not evidence of eldritch forces at work. Yet, scattered across the long history of English league football, a few pairs of teams have met often enough, and still produced zero away wins, that bogey grounds become statistically real.

In the Premier League an example of this is Fulham, and their trips to Arsenal.

Fulham have played 32 league matches away at Arsenal. They have never won.

Not once. Not in 1914 (lost 2-0). Not in 1964 (2-2). Not in 2014 (lost 2-0).

Zero wins from thirty-two attempts. Seven draws. Twenty-five defeats.

Even more striking, this isn’t just a bad example, this is, among every club in the entire Football League dataset, over 135 years of football, no team has a larger, more emphatic, more mathematically convincing record of away futility than Fulham at Arsenal.

So let’s take a look, not just at the record itself, but at how we can quantify bogeyness in a way that recognises your intuition (“two games aren’t enough!”) but takes advantage of the huge dataset football provides.

Why “never won” isn’t enough (the curse of small \(n\))

Imagine you flip a coin twice and get two tails. Does that mean the coin is biased?

Of course not. You need more flips before you start thinking that someone is up.

The same goes for football. A team losing its only away visit to Manchester City tells you nothing. Losing twice? Still nothing except a faint sense of déjà vu. Losing five times? Now you’re paying attention. Losing ten times? Stop buying tickets to the away game.

So what we need is a way of answering this question:

Given a team has played \(n\) away matches at a ground and won none, what is a reasonable upper limit on how good their true chance of winning there might be?

Fortunately statistics, as it always does, gives us a handy tool for working this out, the Wilson confidence interval.

A (gentle) introduction to the Wilson interval

Don’t panic; no equations are necessary. Just an idea. A Wilson interval takes two bits of information:

the number of games played (\(n\)), and
the number of games won (0, in our bogey-ground case)

and asks:

“If their true underlying chance of winning were \(p\), how big could \(p\) realistically be before the observed data (zero wins) would start to look implausible?”

For small \(n\) (not many games played) the answer is: “\(p\) could still be quite large.”

For large \(n\) (a large number of games played) the answer is: “\(p\) must be quite small.”

This lines up with our intuition.

A team with 0 wins from 2 tries? Their true win rate might easily be 30%, 40%, even 50%.

A team with 0 wins from 32 tries? Now the interval collapses. It tells you the true away win probability is almost certainly very small, low enough that even by bad luck alone, you’d be surprised not to have picked up a single win after that many attempts.

The Wilson interval is good for this particular problem (compared to other approaches like the Wald Interval) because it behaves properly at the boundaries (like at 0% or 100%), where ordinary methods can breakdown. Football fans should love it because it gives scientific legitimacy to something they’ve always known: some bogey grounds are imaginary, but a few are real.

The most convincing bogey of them all: Fulham at Arsenal

So, what does Wilson say about Fulham’s 0 wins from 32 away league games at Arsenal?

It says this:

In each of Fulham’s away matches against Arsenal their probability of winning must have been (at the very most) 10% for 0 wins out of 32 to not be statistically suspicious.

Mathematically this comes from plugging \(n\) (number of games) and \(z\) (which is equal to 1.96 for 95% ‘confidence’) into the equation for calculating the upper bound of the Wilson interval:

\[\mbox{Wilson Upper Bound} = \frac{z^2}{n+z^2} = \frac{1.96^2}{32+1.96^2} = 0.107 = 10.7\%\]

So the only way we could avoid saying Arsenal is Fulham’s away team nemesis is if we believe Fulham, across the whole history of the fixture (starting in 1913; see my head-to-head tool here), have not had better than a 10% chance of winning those games. Whilst we can’t be (mathematically) certain that it is true, it is incredibly unlikely. Typical away win rates in the top division over history are roughly 25-30% range. Even clear cut underdogs often have pre-match win probabilities around 15-20%.

So, therefore, I think it is safe to say to conclude:

Fulham away at Arsenal is not just a bogey ground, it is the bogey ground.

The Premier League’s most mathematically defensible curse.

Other bogey teams

Fulham–Arsenal is the only pair in the entire database whose Wilson upper bound hovers at just above the 10% threshold. But several others get close, here are the best (or worst) of the rest:

Away Team	Home Team	Played	Record	Wilson Upper
Fulham	Arsenal	32	0W–7D–25L	10.7%
Grimsby Town	Blackburn Rovers	28	0W–9D–19L	12.1%
Mansfield Town	Reading	26	0W–5D–21L	12.9%
Tranmere Rovers	Barnsley	26	0W–11D–15L	12.9%
Oldham Athletic	Charlton Athletic	25	0W–11D–14L	13.3%
Newport County	Luton Town	24	0W–7D–17L	13.8%
Coventry City	Preston North End	24	0W–9D–15L	13.8%

These are not small samples. These are not casual coincidences. When you are approaching 20 or 30 visits with no victory, it is time to stop thinking you are unlucky and start accepting you have a bogey ground.

Why Our Next Big Solar Storm Warning Could Still Be a Guess

2025-11-30T19:30:00+00:00

Eighteen months ago, a beautiful auroral display swept across the world, accompanying the first extreme geomagnetic storm in more than two decades. It was a landmark event not just for the spectacle but for what it revealed about our growing global dependence on reliable space-weather forecasting. When another storm drew headlines this month (November 2025), with an early “G5 watch” issued by forecasters, governments and operators activated contingency plans in anticipation of another major impact (the geomagnetic storm scale, G-scale, ranges from one to five, with five denoting the most extreme events). Yet the storm ultimately peaked at only G3, a level we expect hundreds of times in a solar cycle. That mismatch exposed a persistent weakness in how we observe and interpret the Sun.

The November events began with three coronal mass ejections (CMEs) erupting from the same active region over consecutive days. Using direct, head-on observations from the Sun–Earth line, scientists estimated their speeds and trajectories and issued a G4 storm watch for 12 November. When the first CME struck earlier than expected and with stronger parameters than the initial estimate, the risk escalated: successive CMEs often travel faster along the newly cleared path, and the potential for a combined impact grows. A G5 watch was released, triggering international preparedness actions. Power-grid operators revisited protection settings, communications providers reviewed continuity plans, and governments braced for potential disruption.

Then nothing happened, at least not at the predicted intensity or time. The CME did arrive, but just before midnight, and with a magnetic field that was only briefly and weakly aligned in the southward direction needed to strongly couple with Earth’s magnetic system. Without that sustained southward field, even an otherwise fast and dense CME cannot deliver a major geomagnetic storm. The result was a modest G3. This forecasting gap did not arise from poor judgement but from a fundamental observational void: after we watch a CME leave the Sun, we lose sight of its evolution for roughly 99% of the journey to Earth.

The limitations of this “single viewpoint” forecasting have been known for years. We can estimate initial CME speed, but we cannot track how it deforms, interacts with the solar wind or merges with other CMEs en route. Crucially, we cannot determine its magnetic structure, the key factor in storm severity, until the CME reaches the L1 point (Lagrange point 1), only about 30 minutes before Earth impact. Even this narrow window is presently fragile: the DSCOVR spacecraft, one of only two satellites providing L1 measurements, is offline, and ACE is approaching 30 years in space with ageing instruments.

Given the rising global reliance on vulnerable infrastructure: power networks, navigation systems, satellite constellations, living with this degree of uncertainty is no longer tenable. Forecasts that overshoot create costly false alarms; forecasts that undershoot risk real damage. The November mis-forecast was harmless, but it underscored how close we are to the limits of what current assets can deliver.

New capabilities are on the horizon, though they require sustained commitment. The NOAA–NASA SWFO-L1 mission is expected to restore robust solar-wind monitoring from mid-2026, stabilising the 30-minute warning baseline. Far more transformative would be the European Space Agency’s Vigil mission, planned for launch to the L5 Lagrange point in 2031. From that vantage point, Vigil would view CMEs from the side, enabling reliable speed and trajectory estimates and dramatically reducing arrival-time uncertainty. But its path to launch depends on political support through multiple rounds of ministerial scrutiny.

Beyond L5, deeper innovation is needed. A constellation of sensors placed in distant retrograde orbits (DRO) could offer early in-situ magnetic-field measurements, potentially hours, not minutes, before a CME reaches Earth. ESA’s planned demonstration of a DRO spacecraft in 2026, HENON, is a first step, but only that. These observational advances must be matched by improved modelling of CME structure, solar precursors and the complex interactions that determine whether an incoming CME will actually be geoeffective.

Accurate space-weather forecasting will always involve uncertainty, but it need not remain dominated by blind spots. The November storm was a reminder that our current system can predict the possibility of major events yet cannot confirm their true character until it is almost too late. If we want forecasts that are genuinely actionable, capable of guiding grid operators, satellite controllers and governments with confidence, then investment in new vantage points, new missions and new science is essential. Without it, our next “G5 watch” may leave us waiting once again, unsure whether a major storm is coming or whether the sky will stay quiet.

40 points to avoid relegation?

2025-11-08T23:04:00+00:00

The number has become part of Premier League folk law. Forty points. Reach 40 and you can relax, the trapdoor to the Championship won’t open.

At the start of the 2015/16 season (the season Leicester City won the League!) their manager Claudio Ranieri set them the target of reaching 40 points to avoid relegation. When they hit the target:

“We have 40 points which was the goal. It’s champagne for my players!”

Claudio Ranieri, 2 Jan, 2016.

But where did this number come from, and it is the right target?

Many people have noted that the 40 point mark is a “myth” (e.g. The Premier League, BBC, The Athletic). But those articles are really just saying that, on average, you need less than 40 points to survive (and the number of points you need seems to be decreasing).

The plot above shows the number of points required to stay in the top division of English football since 3 points for a win was introduced in 1985 (this includes before the Premier League started in 1992). The league tables were created using my arbitrary league table generator. It is also worth noting that during the time range there have been a varying number of teams in the top division (between 20 and 22) which obviously impacts the points required. We have normalized this to a 38-game season for comparison.

A few things are immediatly obvious:

Most of the time you do not need 40 points to avoid relegation (34/44; 77%),
The average number of points needed to avoid relegation over the last 40 years is 37 (in the Premier League era, it is 36 points),
There is a clear decreasing trend of the number of points required.

So where does the idea that you need 40 points come from? There is no obvious origin of the phrase. My best guess is that in the mid 90s, after Southampton survied (just, on goal difference) with 38 points in 1995/96, followed by Coventry City with 41 the following year, then Everton with 40 and then Southampton (again) with 41 points in 1998/99, that that run of four was enough for the (rough) number to stick.

But despite the previous analysis (by me and many others) I think 40 is still the correct, pre-season, target. Moving to (for example) a 37 point target would only give you slightly better than a 50-50 chance of staying up (54.5%). Perhaps you would like a little more certainty…

One way to look at this is to calculate the cumulative distribution function (CDF). The CDF is a function that shows the probability that a random variable is less than or equal to a specific value. It “accumulates” or “adds up” the probabilities for all outcomes up to a certain point.

The above figure shows two, slightly different, CDFs, in blue across the whole dataset (since 1985) and in red just in the Premier League era. To read the plot look at the number of points on the x-axis and then read off the corresponding probability of avoiding relegation on the y-axis for that number of points.

Now the 40 point value makes a lot more sense, as it gives a team over an 80% chance of staying in the division, now I prefer those odds!

So, whilst there are plenty of posts online telling you that the 40 point target is a myth, I think, as pre-season target for teams, it is still a good one.

New space weather modelling suite enables upper atmosphere forecasting

2025-10-08T14:31:00+00:00

A pioneering new space weather forecasting modelling suite will enable operational modelling of the upper atmosphere at the Met Office for the first time in a major breakthrough for UK atmospheric science.

The Advanced Ensemble Networked Assimilation System (AENeAS) is a new suite of space weather forecasting models available to the Met Office that focuses on how space weather can influence the thermosphere and ionosphere here on Earth.

The suite, built at the University of Birmingham, and developed in collaboration with Lancaster University, the Universities of Leeds, Bath and Leicester and the British Antarctic Survey is now running on the Met Office’s new supercomputer.

The deployment of this suite at the UK Met Office is the realization of a 10-year vision of SERENE, to build and deliver a state-of-the-art upper atmosphere modelling capability into operational use. The tools will be able to support a wide range of users and ultimately allow people to make informed decisions earlier, being proactive rather than reactive in their response to space weather.

Professor Sean Elvidge, Head of Space Environment and Radio Engineering (SERENE) - University of Birmingham

Complementing the Met Office’s existing space weather forecasting models, which include predicting the arrival of events from the surface of the Sun, this system introduces new forecasting capability for modelling impacts from space weather on satellites, aviation, communications and services which rely on GNSS.

Professor Sean Elvidge, Head of Space Environment and Radio Engineering (SERENE) at the University of Birmingham, and the lead developer of the system said: “The deployment of this suite at the UK Met Office is the realization of a 10-year vision of SERENE, to build and deliver a state-of-the-art upper atmosphere modelling capability into operational use.

“The tools will be able to support a wide range of users and ultimately allow people to make informed decisions earlier, being proactive rather than reactive in their response to space weather.”

The new suite means that, for the first time, forecasters at the Met Office Space Weather Operations Centre (MOSWOC) will have access to forecast model output on the impacts of space weather on the ionosphere, as well as enhanced modelling of the thermosphere.

Once again, cutting-edge British innovation is making a remarkable difference to our daily lives - this time from way up in the atmosphere. This is a really exciting example of how better understanding of what’s happening in space can protect the tech we all rely on, from GPS on our phones to keeping the power grid working.

UK Science Minister Lord Vallance

Met Office Space Weather Manager Simon Machin said: “This delivers a world-leading capability that provides greater confidence and forecasting skill than any models currently in operation anywhere else in the world.

“This isn’t just about science - it’s about protecting the systems we rely on every day. From aircraft communications to GPS in your phone, space weather can affect us all.”

Science Minister Lord Vallance said: “Once again, cutting-edge British innovation is making a remarkable difference to our daily lives - this time from way up in the atmosphere.

“This is a really exciting example of how better understanding of what’s happening in space can protect the tech we all rely on, from GPS on our phones to keeping the power grid working.”

The new modelling capability will be able to assimilate near real-time data about the current state of the ionosphere and thermosphere, combine with forecasts of solar activity from the Sun to produce accurate and actionable forecasts of the upper atmosphere. This will help enable service providers to take mitigating actions to prevent impacts from space weather where possible.

Professor Farideh Honary from Lancaster University said: “We are happy to see our research being translated into a useful product to be used by industry. The research and modelling led by Lancaster is relevant to the aviation industry and in particular to flights using polar routes which are dependent on high frequency communications.

“These flights have significantly increased since their initial opening in the 1990s due to their operational advantages such as reduced flight times and fuel consumption, which translate to cost savings and environmental benefits like lower carbon emissions.”

More accurate and precise forecast information will help enable service providers to take mitigating actions to prevent impacts from space weather where possible. One example of this is with users of Global Navigation Satellite Systems (GNSS), such as GPS, for positioning or navigational purposes. If they understand that there is likely to be a loss of accuracy in GNSS, they can switch to using other systems.

Together, the new modelling suite has been delivered as part of Space Weather Instrumentation, Measurement, Modelling and Risk (SWIMMR) programme, which was funded through the UKRI Strategic Priorities Fund and designed to enhance the UK’s capability for monitoring, modelling and forecasting space weather.

Professor Ian McCrea, SWIMMR Programme Lead at STFC RAL Space, said: “These new models represent a significant step forward for the UK’s capacity to model, forecast, and understand key components of our upper atmosphere. By coupling advances in physical modelling with global scale observations, they will enable unparalleled awareness of Earth’s geospatial environment.

“The models promise to address a wide range of use cases, ranging from radio communications to the prediction of satellite orbits, and we expect that they will be of great importance to a huge variety of stakeholders. They also provide a clear demonstration of how the collaboration between academics and end users, which SWIMMR has enabled, can benefit members of both communities.”