@joshuabrooks said in Vintage Challenge - 8/5/17:
When dealing with sample sizes of 30-50 people, is it starting to become more of an issue of what decks the elite and non-elite players choose each week, or do you think these numbers are an accurate representation of each deck's relative strength to the field and enemy decks?
The MTGO metagame is just like any other metagame. There are a handful of players that play in nearly every event and do well, which can certainly skew the results. Throw in all the other unique characteristics that exist between different metagames and the online meta (card availability, player population/preference/competitiveness, etc.) and you have to regard any data from this metagame in the proper context and limit the scope of the conclusions derived from it.
That said, we really aren't inundated with data when it comes to the Vintage format. What we have from the large events (which are likely the best sources of data) correlates pretty closely with data from Challenges.


Shops 61.3% to 60.5% and Gush/Mentor 50.2% to 52.6%. Granted, this is before vs after the Gush restriction, but I still think it can be argued that MTGO is a reasonable approximation of the broader Vintage format. Now, I certainly wouldn't look at events in isolation. But I also would consider that the Online metagame is actually pretty large as far as the available pool of players. From the May to July sample, 202 unique players participated in the Challenges. I doubt there are many if any regional metagames that draw from a larger pool for comparably sized events.
I understand this is a bit of a tangent from what you asked but I thought it was an opportunity to bring it up. As far as it being an accurate representation of a deck's relative strength, it's really not perfect for many reasons (those mentioned earlier and others). But the alternative is people's subjective opinions based on perceptions from their own testing groups or regional metagames. So while it's not incredibly accurate and individual players can have significantly different results, I think is the most accurate objective measure available. We unfortunately don't have anything better.
Do you think the data accurately reflects the matchups from a pure point of view (ie if it was a pro playing a pro?) Just curious about your opinion before this devolves into another B/R discussion 
This is a hard question to answer. The objective results from the events match my subjective views on the format, which is I think the spirit of what you are asking. Shops really does feel like the best deck and not just due to a lack of hate cards. Big Blue, Blue Control, and Mentor feel much closer in power level following the Gush restriction. While a pro might have a higher win rate with certain "skill-intensive" decks, it don't think the effect would dramatically shift the current hierarchy in Vintage.
Hope that answered your questions, Josh.