Today's 40k Metawatch Article was a disappointing start. Here are some of our concerns.
Author: Cliff Thomas
Today, Warhammer Community featured a Warhammer 40 Metawatch article describing their understanding of the overall competitive meta. While it's encouraging to hear that they'll be returning to a more consistent publishing schedule, and very exciting to hear that the new dataslate is incoming, we came away overall disappointed. This was due to a few things - first, a few statements that were demonstrably false in the competitive dataset. There were also some very broad statements about the relationship between win rates and player experience that lack nuance, and are again outright false in our data. Last, we have very real concerns about their articulated approach - one which focuses overwhelmingly on overall faction win rates - to make balance changes. Let's get into some of their statements.
“Therefore, we afford a margin of error of 5% either way in the data we collect, and aim for each army to show win rates between 45% and 55% – if it’s higher, it suggests the faction’s overperforming relative to its rivals, and if it’s lower it suggests they’re struggling to keep up…
…Most of the factions fall in that sweet spot which suggests that overall the armies – and the game – are pretty well balanced. A few find themselves on either side – there are several reasons for this, and a number of things we can do to influence those results.”
Overall, this statement rings true. A bit of reframing gives us a different answer though - in our dataset 36% of games played were by factions falling outside the Goldilocks zone. That's a lot of games happening outside the desired balance state!
“You’ll see that Adeptus Astartes armies (those made using Codex: Space Marines) fall into the underperforming category, with a win rate of just 39%. Space Marine armies are incredibly prevalent, as the iconic armoured face of the 41st Millennium. They’re often new players’ first army – and it’s fair to say that most people aren’t especially gifted at the noble art of Warhammer 40,000 when they first start playing. It also means most players are quite familiar with them – and how to play against them. This is part of why their results don’t quite match their indomitable reputation.”
Ehhhh. This is very hand-wavy. There are some power-armoured factions (like Blood Angels), that are doing well enough - the Sons of Sanguinius are sitting at a 47% Win Rate in our dataset, and Iron Hands are at a 45%.
“Although each army has its own characteristic fighting style, certain features are common to several factions. Space Marine Chapters, the various Traitor Legions, the Adepta Sororitas, and even the Necron dynasties have a great deal setting them apart from one another, but they can all call upon a variety of squads equipped with a 3+ Save and keep foes at bay with boltguns or other medium-range weapons.
Such forces are sometimes referred to as MEQ (Marine Equivalent), and there are tactics that apply equally when commanding or combating such armies.”
If the above was true in the broad sense, then we would expect the win rates of the Astartes factions, Chaos Space Marines, Sisters of Battle, and Necrons to be mostly aligned, with a few outliers here and there with drastically different play-styles due to sub-faction rules. That is clearly not the case, at least not in GT+ (5+ rounds, 25+ players) events:
“Factions like the Harlequins or Genestealer Cults, on the other hand, play very differently, with units and special rules that are harder to compare. As such, they have more of a learning curve, rivals are less familiar with them, and the players who master them tend to be more experienced than most.”
Is this true? We group each of the players in our dataset by their attendance at GT+ size events. In nearly all faction cases, players who are attending their very first event make up the plurality (and occasionally the majority) of players who have played at a GT+ event in the Nephilim meta.
If we use event attendance as a proxy for player experience, then it's pretty clear that experience matters. This isn't surprising! That said, we've built our dataset to allow us (and you all) to see how players with equivalent experience match up against one another - if outcomes for factions across these experience brackets look roughly similar, then we should know that factions consistently landing outside of the Goldilocks Zone are in need of a look.
In short, that's what's been the case for the factions cited above in both Nephilim and Nachmund. for the two examples GW used - Harlequins and Genestealer Cults - win rates by Casuals into other Casuals have been incredibly high and quite fine, respectively. Throughout Nachmund, Casual Harlequin players posted a 67% win rate, and in Nephilim they're sitting at 56% (that rises to 61% for the Light subfaction). That's really high for first-timers! Genestealer Cults doesn't show this level of performance - they've been between 45% and 55% for all iterations of the GT+ Meta since Nachmund's release.
In short - Games Workshop can't use the excuse of an experienced player base to explain away the performance of a faction like Harlequins (or release Custodes / Tau, or Tyranids…). Doing so appears to be wilfully ignorant of the data that's publicly available on faction performance, and risks serving as the basis for changes that are sub-optimal.
There are a couple of larger concerns here that the article alludes to. First is that they're missing a lot of nuanced data - TiWP, over-representation among top 4s and winners across events, disparities in performance between experienced and inexperienced players, patterns that they may be noticing in RTT vs large event play, record distributions across factions, secondary choices and scores, etc. If they're tracking this data and using it to inform balance changes, they should transparent about their process. Unfortunately, I don't think that's the case - it really does seem like they're just using raw faction win rate as the primary balance consideration.
That's unfortunate, because overall win rates hide a lot of nuance. Using overall faction win rates as your sole consideration will inevitably lead to the wrong adjustments. At minimum, sub-faction performance data would give a more accurate starting place for changes. Going back to Harlequins as an example:
The Light Saedath is what's driving Harlequins over-the-top event performance. Any series of nerfs that applies to the faction on the whole will hurt performance for Twilight and Dark Saedath players down to potentially sub-optimal levels - they would be bystanders in a shotgun approach to a balance pass. A more specific look at the Light Saedath (like maybe their ability to advance and shoot at no penalty with highly mobile gun platforms and troop models???), is likely to yield a better outcome for the faction on whole.
On the other end, average overall win rates can conceal over-performance from a sub-faction. At first glance, Chaos Space Marine's 49% win rate would make it seem as if they’re nearly perfectly balanced. However…
There's WIDE variation in the performance of the various traitor legions. Creations of Bile are clearly in need of a look (likely a nerf to fight on death), while World Eaters and Iron Warriors could use a helping hand.
Overall, this was a fairly disappointing article. Given our data-driven approach, we were hoping that the 40k Design Team would articulate an approach similar to what we saw from the Age of Sigmar team. While imperfect, the AoS rundown was much more clear about their design intent, and did a good job explaining some of the intricacies we've covered above. Overall their analytical approach was much more similar to what we've seen from the The Honest Wargamer's AoS Stats Center. That's a good thing - the AoS Stats Center crew does great work! The Warhammer 40k team should be taking notes from their colleagues across the hall, and drawing resources (and inspiration!) from the analysis that we here at Stat Check, Goonhammer, and others are already providing. Our internet door is always open.
Here's to hoping for a Corvus Blackstar buff in the upcoming dataslate. Happy hobbying, and good luck gaming!
