Warzone Atlanta: Crunching the Numbers

In a previous post, I went over Warzone Atlanta from the perspective of an event attendee. Now that I’ve gotten my hands on the data from the event itself, I thought I would take a look at the event from a statistical perspective. I’m also trying some new techniques analyzing tournament data, with…moderate success.

Data: The data used in this analysis was kindly provided to me by the tournament organizers for Warzone Atlanta for both 2015 and 2016, but it’s little more than what should be available publicly on Best Coast Pairings when the results are posted. WZA is a decently large tournament, but there are, as usual, some sample size issues. I’m working on resolving some of those for the future by more aggressively pooling tournaments, but for now, it is what it is. I did exclude one entry from the analysis because it was missing faction information.

Army Representation:

The figure below is a bar chart of all the armies, showing the total numbers that came from each army:

No surprise, there are a ton of Eldar armies, and a decent number of Space Marines of both the Chaos and Loyalist variety. Compared to the LVO, which I’ve analyzed in the past, Space Wolves and Khorne Daemonkin are pretty over represented, and while frequent, there are less overall Space Marine armies, and far fewer Necron armies proportionately. There are a number of possible explanations for this – the fading of Gladius as a power list, or the higher number of allowable factions meaning Space Wolves are more popular as a foundation for “A Little Bit Of Everything” Imperial armies. Relatively new armies, like Deathwatch and Genestealer Cults are rare, as are some struggling armies, like Harlequins.

Though again, the one Harlequin army there was beautiful.

Now, onto the results.

Overall Army Results: The overall median score for the event was 84 points. The mean was 85.42. For the sake of brevity, and because there’s going to be a long list later, I’m just going give the visual results of the tournament, instead of going over each faction’s media. Note armies with a single entry have been excluded here.

This is, to me, a very encouraging plot. There’s a much less clear tiering structure here than there are in most other tournaments – there’s some obvious standouts driven by high placing wins, but far less runaway factions. For example, Chaos Space Marines are literally all over the map, and only the Tyranids are strongly below-average performers. A few other highlights:

• Of the Space Marine factions, Dark Angels appear to be the top performers at Warzone Atlanta.
• The Necrons did surprisingly poorly. In the last couple tournament analysis posts I’ve done, they show up as a top tier army. Not so here.
• The Eldar, while still quite strong, aren’t the clear dominant faction at WZA.
• The Tau…did really well. In most tournaments, I characterize the Tau as a reliable way to get a middling-good finish, but not a great way to end up at the top tables. At WZA, that distribution is way skewed upward. I’m going to attribute this almost entirely to the Ta’unar Supremacy Suit being allowed in this event. Whether this is desirable or not is open to debate, but this is a clear illustration of how one’s tournament rules and army comp can profoundly influence the game. These are not little decisions. I also wouldn’t necessarily bank on Forge World fixing this unit soon (given the fairly slow pace of releases). But this also means you should take musings on armies with a grain of salt until you understand the local meta a commentator is coming from.

Multipliers:

In my “What’s Wrong With the Tau?” article awhile back, I provided something I called a score multiplier for several armies. Basically, this is how many points above average one can expect to do playing Army X vs. all other armies. For example, if an army’s multiplier is 1.50, they are expected to get 1.5X the battle points of an average army – in the case of WZA, that would be ~ 128 BP. What follows is a table of all the armies at WZA with more than one entry, along with a measure of uncertainty (a 95% confidence interval for those who have taken stats). Basically, an army with a number greater than 1 performs better than average, and an army with a number less than one performs worse than average.

• Blood Angels: 1.03 (0.76, 1.38)
• Chaos Daemons: 1.11 (0.88, 1.42)
• Chaos Space Marines: 0.91 (0.74, 1.11)
• Dark Angels: 1.04 (0.81, 1.32)
• Dark Eldar: 0.90 (0.59, 1.37)
• Eldar: 1.15 (0.99, 1.35) – this goes to 1.19 (1.02, 1.39) if you exclude the author of this post, also known as the Worst Eldar Player.
• Imperial Knights: 0.88 (0.67, 1.14)
• Khorne Daemonkin: 0.93 (0.76, 1.14)
• Necrons: 0.88 (0.67, 1.15)
• Tyranids: 0.66 (0.47, 0.93)
• Orks: 1.10 (0.78, 1.54)
• Space Marines: 0.83 (0.69, 0.996)
• Space Wolves: 0.96 (0.80, 1.15)
• Tau: 1.22 (1.10, 1.36)

Some notes on this – in the WZA meta, the Eldar and Tau are the top performers overall, with Chaos Daemons trailing behind them, and most other armies clustered around 1. This is, again, not a bad thing. A perfectly balanced game has everyone around 1. This does mean I may give the guys on Forge the Narrative a break for complaining about Tau ruining the game – in their area, they are. Though in fairness, it’s also the logical consequence of their very free ranging composition rules.

Predicting Performance:

To extend the analysis of WZA a little past what I’ve done for other tournaments, I tried out the idea of using these multipliers to predict the performance of each individual player. I was hoping to use ITC points as a means of prediction as well, but a large number of players only had WZA as their ITC result. Instead, I settled for their previous placing at the 2015 WZA, with players who didn’t attend last year assigned the mean placing (there are other ways to handle missing data like this, but I think this is a reasonable first pass).

The score multiplier for previous placing is 0.992 (0.989, 0.996). So basically, for each one unit increase in your placing in 2015 (First to Second, or 27th to 28th) your predicted BP score in 2016 is reduced by a little under 1%.

So then…how does something like this do?

The red line would be perfect performance – the predicted score perfectly matching the actual score for each player. Clearly, this model isn’t perfect. There are a number of reasons for that, each of which could stand some refinement:

• There are just bad players. For example, myself. My result was 57 BP, and my predicted result was 75. The model cannot yet distinguish things like “This guy brought a friendly list to a major tournament”. This really can’t be fixed, though in heavier ITC-focused environments, using ITC points might help.
• Similarly, it’s difficult to detect good players. Of the Top 10 players, 3 hadn’t attended WZA 2015, which dragged their predicted scores down. This is, if nothing else, the inevitable consequence of the growth of the event. As event sizes stabilize, this may improve.

Basically, a model built largely on just “what army do you play” actually turns out to underestimate the performance of players – clearly more work is needed in trying to quantify player ability as well as army performance. Right now, the model just can’t capture top players – it underestimates about 30% of the field. But it’s a promising first step.

Closing Thoughts: Despite the massive changes we’ve seen in the game recently in terms of the ruthless release schedule, there’s not a huge amount of disruption yet. There are, for example, not enough Genestealer Cult armies operating yet to understand if they have a major effect. But what we can see comparing this to other tournaments is that it acts as a very vivid illustration of how impactful army composition rules are. There’s not enough data here, yet, to see about other effects, like the WZA mission set. Perhaps someday.

In the meantime, my plan is to start more extensively pooling tournaments to get large sample sets, and thus, hopefully, start improving predictive models of tournaments. Stay tuned.

Enjoy what you read? Enjoyed that it was ad free? Both of those things are courtesy of our generous Patreon supporters. If you’d like more quantitatively driven thoughts on 40K and miniatures wargaming, and a hand in deciding what we cover, please consider joining them.