My intention was to write just one piece on this subject, but as it swelled it became apparent that splitting it in two was for the better. Part two can be found here.

A couple of weeks back there was a discussion on Twitter regarding the criticism that Liverpool goalkeeper Loris Karius has received from the English football media. I’m a particularly selective reader when it comes to football, so I hadn’t given the criticism much attention but I saw some WhoScored infographics floating around showing Save%, and how both Karius and Claudio Bravo look pretty mediocre according to that statistic. Analytics Twitter quickly jumped to their defence, protesting that since we’ve developed our understanding of the game enough to not treat every shot equally from an attacking point of view, why should we do so when looking at goalkeepers?

Fair point, I think. So why not use Expected Goals instead? Well, as was quickly pointed out, it’s not so simple. There was criticism because it seemed to be something of a reverse engineering of the metric. There was also criticism because conceding goals is practically never only the fault of the goalkeeper, so therefore applying a metric that is unable to account for defensive pressure is bound to struggle to capture the true skill of the goalkeeper, or even to separate it from the skill/organisation of his defence.

That being said, I still think it’s the best we’ve got (and I’m not alone in thinking that). Analysing goalkeepers is always going to be very contextual, and because goalkeeping is such a multifaceted skill, measuring it is always going to take more than any one metric can capture. Then again, that’s true of pretty much every other position as well, so one shouldn’t be discouraged to at least give it a go.

The thought brewed for a while, and then Otso Virtanen signed for KuPS a couple of weeks ago. Virtanen had a breakthrough season in 2015, playing for IFK Mariehamn which got him a move to Hibernians, where he only started one game. His return brought with it some consternation from fans of IFK Mariehamn and a boast from new KuPS manager Jani Honkavaara that he’s the best goalkeeper in the league. Now, this is far from a controversial statement. After all Virtanen certainly did something to impress the people at Hibs. And even if he only played for them once, he’s still only 22 so the experience will have done him a world of good in any case. Most keepers that young sit on the bench or play for the reserves, so at the very least there likely wasn’t any harm done.

But uncontroversial doesn’t equal true, and even if the metrics available aren’t perfect to conclusively determine the absolute quality of a keeper, they’re certainly good enough to do some comparisons. In other words, it’s time to check some facts.

I’m going to look at goalkeeping from four different angles, two in this piece, and two in Part Two. I’ll start off by looking at goalkeeping from the perspective of cumulative Expected Goals values, then I’ll try to look more in-depth at the difficulty of the shots leading to goals, followed by a look at saves and finally I’m going to try to quantify how well the goalkeeper controls his zone aerially – something that doesn’t show up in Expected Goals calculations because it’s a way of preventing shots rather than saving them. In the end I’ll hope to have something approaching an answer to the question of who the best keeper in the league might be.

So, I set out to find out what a simple ExpG – Goals conceded stat would look like before I quickly realised that if anything it makes even less sense than Save%, because now you’re rewarding goalkeepers for things they have nothing to do with – blocked shots, shots off target etc. In order to truly capture the quality of a goalkeeper from a shot quality standpoint, you need to only look at the shots that hit the target (something that’s talked about here). This is quite doable yet simultaneously somewhat problematic. I’ve previously noted that the sample size that I’m working with isn’t quite sufficient – well, strip away 64% (shots off target and blocked shots) and it certainly doesn’t help.

That’s just something we’ll have to contend with at this point, which is to say that the following is somewhat experimental. I reworked my ExpG-model so that it’s based on shots on target, and due to the decrease in sample size I also stripped away everything else apart from shot location. This means that where regularly an average shot would have a goal expectancy of around 0.10, since we’re only counting shots that hit the target, that figure changes to around 0.30.

I also removed penalties from the equation completely to get a more general idea of goalkeeper quality. I think there is an argument for not removing penalties when talking about goalkeepers. The argument for removing penalties from outfield player shot counts is that they skew Goal, ExpG and conversion numbers in favour of the designated penalty taker, as they are essentially unevenly distributed ‘free’ quality chances. Since goalkeepers similarly cannot choose whether to face a penalty or not it feels like the skew is less of an issue for them. That being said, since penalties are pretty rare (about 4 per team last season) and penalty conversion is pretty high (about 85% in the Veikkausliiga), it isn’t at all weird to see a goalkeeper not save a single penalty in a season which in turn creates a situation where his numbers might be affected by something that requires a completely different expertise than what we’re trying to measure here.

I also removed own goals conceded, because they usually aren’t the fault of the keeper, and because they don’t register an ExpG value.

Ok, so with that out of the way, let’s look at some names. I’m using data from the past two seasons with 900 minutes played as a cutoff point.

Player season GKPerformance: Save%
Kreidl 2016 (KuPS) 8,96 83,0%
Viitala 2016 (MIFK) 6,74 80,0%
Kobozev 2016 (VPS) 4,35 77,6%
Aksalu 2015 (SJK) 3,92 81,0%
Schuck 2016 (RoPS) 3,81 79,6%
Schuck 2015 (RoPS) 3,79 81,4%
Sillanpää 2015 (VPS) 3,16 74,8%
Maanoja 2016 (Lahti) 3,11 73,3%
Sahlgren 2015 (KTP/HJK) 3,04 80,4%
Bahne 2015 (Inter) 2,65 77,6%
Örlund 2015 (HJK) 2,59 77,2%
Reguero 2016 (RoPS) 2,41 75,3%
Chencinski 2015 (RoPS) 2,10 77,4%
Lehtovaara 2016 (Inter) 1,44 75,6%
Vilmunen 2016 (PSK) 0,87 73,1%
Dähne 2016 (HJK) 0,79 73,0%
Vasyutin 2016 (Lahti) 0,75 71,7%
Eriksson, Carljohan 2015 (HIFK) -0,13 69,4%
Hilander 2016 (Ilves) -0,68 71,0%
Virtanen, Otso 2015 (MIFK) -0,82 76,7%
Pyhäranta 2015 (KTP) -1,65 73,4%
Moisander 2015 (Lahti) -1,73 70,5%
Maanoja 2015 (KuPS) -1,87 74,4%
Johansson 2015 (HIFK) -2,38 67,5%
Pennanen, Juhani 2016 (PSK) -3,86 68,6%
Eriksson, Carljohan 2016 (HIFK) -4,24 67,5%
Aksalu 2016 (SJK) -4,33 73,8%
Kollar 2016 (PK) -4,54 67,6%
Öst 2015 (Jaro) -4,91 67,6%
Hilander 2015 (Ilves) -5,18 71,2%

With this table I wanted to compare a goalkeeper performance metric based on Expected Goals with Save% to see whether the results are more or less similar. Goalkeeper performance is calculated by adding the Expected Goals of every shot that the goalkeeper has either saved or conceded a goal from and then subtracting the amount of goals he conceded from those shots. Basically, first we add up how many goals the keeper would have been expected to concede from the shots that he has faced, and then by removing the real amount of goals we get an idea of how many goals he has ‘saved’ above expectation (or ‘conceded’ below expectation).

There is a definite correlation between Save% and GKPerformance, but there are some outliers. I think this is fairly intuitive, as a goalkeeper will have to make a fair amount of saves to make up for the goals he concedes. GKPerformance, however, takes into account some of the information that Save% doesn’t, rewarding a goalkeeper for saving difficult shots while punishing him for letting in easy chances.

So does GKPerformance pass the eye test? Johannes Kreidl and Walter Viitala scored high using my regular Expected Goals model as well, so to see them top the list is no big surprise. Mihkel Aksalu shows up both at the top and at the bottom which is worry from a consistency viewpoint, otherwise repeat entries are more or less clustered. More player seasons are definitely required to be able to determine the reliability of the metric, but it looks pretty decent.

As this piece is, at least partly, about the much heralded return of Otso Virtanen it’s worth noting that in his case, the data says ‘meh’. In fairness, IFK Mariehamn underperformed pretty spectacularly in 2015, but it looks like his performances didn’t help with that. His Save% in 2015 was actually well above average (76.7% compared to 72%), but he conceded about one more goals than would have been expected.

The above performance metric is a decent top level overview of how well a goalkeeper has played, but to get a better idea we need to take a closer look. Determining whether a goalkeeper is good or not from a technique point of view requires a specialised expertise that few have, and doing it through the use of numbers if practically impossible at this point in time, without detailed positioning data. One thing we can do, however, is to strip all of those things that are difficult to evaluate from a goalkeeper’s performance and focus wholly on what he’s looking to avoid – letting in goals. Let’s make the assumption that on a very basic level what you want from your goalkeeper is to save all the easy chances, to make himself as tough as possible to beat. So by taking the cumulative Expected Goals from the shots that he has conceded from and dividing it by the amount of goals he has conceded we can get a metric for how difficult it is to score past him.

Here’s the data in a scatter chart with the above GKPerformance metric as the other axis.


Presented like this, the players can be clustered into four categories depending on whether he has performed better/worse than expected and whether it is ‘easy’ or ‘difficult’ to score against him.


Unfortunately for Virtanen, this chart doesn’t do him many favours either, as it seems like it was easier than average to score against him in 2015. There’s also a separation (albeit minor) between the top two in GKPerformance, since Viitala conceded more difficult chances than average whereas Kreidl was below average.

Based on these two metrics, there’s very little evidence for the notion that Otso Virtanen was the best keeper in the league in 2015. That obviously doesn’t mean that he isn’t any good in 2017, just that maybe there’s some reason to temper expectations a bit, especially when considering the level of performance of Johannes Kreidl for KuPS in 2016. Goalkeepers peak at a later age than outfielders, and Virtanen is only 22, so there’s plenty of room for development.

Then again, if we’re looking for the best keeper in the league, there’s reason to compare to the rest of the crowd, and looking at the names in the upper right quadrant, Tomi Maanoja is the only keeper with a contract with a Veikkausliiga side for next season, and he can also be found in the bottom left one…

That’s it for part one, in part two I’ll be taking a more granular look at the different outcomes of the shots on target that a goalkeeper faces and a look at how well he controls his zone aerially. In addition I’ll try to wrap it all up with some conclusions, limitations and potential areas of improvement.

Follow me on Twitter @Minor_LS


One thought on “Fact checker: Is Otso Virtanen the best goalkeeper in the league? – Or, quantifying goalkeeper performance: Part One

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s