Living in the era of big data and Magic Online, we have more data and performance measures than ever before, prompting some debate on the utility of various measures.
Performance measures we now have include:
Top 8/16 & Daily Event 3-1/4-0 appearances
Metagame Penetration (% of a deck/archetype/card/engine of total metagame)
Matchup Win Percentages
Specific Matchup Win/Loss%ages
We can even add other performance measures, such as average rank performance, mean rank performance, etc.
The goal of the DCI is to promote competitive balance, and to use restriction as a policy tool to maintain competitive balance, just as the FTC and federal courts use antitrust laws to maintain market competition. These tools are most obviously appropriate when there is a dominant deck or monopoly power.
But what's the best measure of dominance or performance?
For most of the history of the format, we lacked detailed overall metagame data. All that we had was Top 8 decklists, if that. Now, with Magic Online, we can see every deck that was played in a Premier event, giving us more data than ever before. We can see the entire metagame and compare it to the Top 8. We can see how players performed, and calculate win percentages. But what does all of this tell us?
Here are some initial thoughts:
1) Performance Matters
It's possible for a deck to have an absurd metagame presence, and yet perform poorly. For example, it is possible for a deck to be 60% of a field, but have 0% of the Top 8 or Top 16, and a sub 50% Match Win %.
2) Matchup Win %ages matter.
They tell us whether a deck has bad matchups or not. They are not the definitive answer, but they tell us something.
3) Performance Over Time Matters
Metagames are dynamic. A deck can dominate a tournament one week, and get crushed the next. Performance metrics at any single point in time cannot be determinative. We must understand how decks perform over time, and dominance demonstrate sustained performance, not performance that waxes and wanes.
I don't pretend to have the answers, but these three principles suggest a few things conclusively, as a matter of logic:
- Metagame Saturation is not a Performance Metric, and cannot be a valid direct indicator of dominance. This metric is only useful as context - in helping us see whether a deck under or overperformed their presence in the field.
The best example of this is the May MTGO Premier event. Eldrazi were 14% of the field, but 63% of the Top 8.
For that reason, I consider metagame saturation the worst possible indicator, and is only a valid indirect/contextual indicator. It is logically incapable, alone, of telling you anything about performance.
- Tournament size matters
Tournaments must be large enough to have enough players so that decks can have a chance to compete and show up in the data. For that reason, I'd give more weight to larger tournaments, and less weight to things like daily events. But within Daily events, I'd give more weight to 4-0 performance than 3-1.