Introduction
I think it’s long past time people stopped taking “Hearthstone Mathematics” (u/HS_Mathematician) seriously, for those that ever did. This person tries very hard to wrap themselves in a veil of mathematical authority, yet doesn’t seem to have even a basic grasp of how statistics or data analysis is actually done, and I seriously doubt they have any formal mathematical training. I can see only two possibilities: they are talking out of their ass, or they know better but don’t care because they have a narrative they are trying to push and just want to get there with the least effort possible.
Bold Claims
This individual recently posted what I will generously call an infographic with the headline “The number of players who are willing to invest money has been reduced by almost half, 33% of players continue to boycott Blizzard. What should "Team 5" do to earn your trust?” https://www.reddit.com/r/hearthstone/comments/k7ujtc/the_number_of_players_who_are_willing_to_invest/
Hard Evidence?
What evidence is provided to support such strong claims? The certainty of the headline suggests hard evidence, like quarterly earnings reports with appropriate loss of revenue and statements from Blizzard promoting that narrative explanation. You won’t find one, and the most recent Q3 2020 report from ActBlizz itself (which it should be noted will not include backlash from the battle pass--we’ll have to wait for Q4 for that, or even Q1 2021), shows growth and an increase in hours played year over year. When you look at virtually all metrics, 2020 has been one of their best years. We have a lack of hard data, and what we do have doesn’t support these claims.
Bad Statistics
However, there are other ways to know things. What we were given are the results from two Youtube polls conducted by Hearthstone Mathematics with the implied comparison intended to show a decline in player purchasing behavior. Any person who has done data analysis knows that there is as much art as science to it. We have powerful mathematical theorems to help us, but they often assume perfect conditions that are simply not met in practice. Considerable effort and years of education and training are needed before a person can confidently employ best practices in regards to administering and interpreting polls. Expertise that I, someone with a formal mathematical education and experience in data analysis, do not fully possess myself because I do not work as a pollster. What is on display from Hearthstone Mathematics is laughable even to me.
There are many problems with this comparison and I’ll go over them now. This is not a comprehensive list, and I welcome others, especially those with more polling experience than I have, to weigh in as well.
1) The two surveys do not ask respondents about their willingness to purchase products from Blizzard in general or whether their spending habits have or will change in accordance with the battle pass, but about their willingness to purchase specific products. Additionally, one survey asked if they were planning to purchase a product while the other asked if they had purchased it already. Small changes in the way a question is asked can change the results, much less asking completely different and unrelated questions. One question asks whether the purchase is “profitable” while the other asks if it is “worth it”; vague and useless.
There are two kinds of pollsters: those who don't know this and make bad polls, and those who do know it and skew their questions to get the results they want. One is negligence, the other malice. A better survey would not have used leading language in the response choices, and would have asked more than one question that determined whether the respondents had spent money in the past, whether they planned to spend money in the future, whether their feelings towards Blizzard or their behavior influenced their decision (which by the way the reasons people were claiming to protest were very different when the two bundles came out as one was about Blizzard’s behavior politically and the current drama is about ingame rewards) to buy products or whether their decision to purchase was based solely on the perceived value of the products, etc. Then, you give the same survey again so that you can get comparable results and look for correlations in the data (which as everyone knows should not be confused for causation, but we didn’t even make it that far, he went straight from nothing to causation).
2) In order to get good results from a small sample size, you need a random independent sample, which is something that all polling suffers from. Selection bias, either on the part of the people administering the survey, or self-selection bias on the part of people who choose to take an optional survey is a big problem. The fact that he only asked his own youtube audience automatically rules out the idea of this being a representative sample. It only covers English speaking players who are engaged enough to seek out content on other platforms, and among those only the ones motivated enough to answer a survey. Hearthstone is an international game with players all over the world, and the game isn’t even the same in every country as we know from the systems designed to circumvent gambling restrictions in China. The culture is different in different parts of the world, and not all players who use social media to discuss or learn about the game use Youtube or watch his channel.
Good polls usually have a margin of error based on a confidence interval, the mathematics of which only hold water if you conform to the assumptions needed to make them work, and while there are many, the independent random sample is high on the list. There is no confidence interval or margin of error given, nor would any such calculation be worth anything when we know the underlying assumptions are not met to make them valid. Good polls struggle with this as it’s nearly impossible to get a truly random independent sample, but they make every attempt possible and in the case of important things like elections, multiple polls are combined with other data into a richer model. Absolutely no attempt was made here and HS_Mathematician even admitted that he made these surveys without any intention of comparing them, showing just how lazy and half-assed this attempt at data analysis was.
3) The conclusions don’t make any sense and are heavily misinterpreted if not outright wrong. He says that “the number of players willing to invest money has been reduced by almost half”, based on the data that 27% of his respondents said they were planning to buy the Felosophy bundle, while only 15% of his respondents claimed to have purchased the C’Thanks bundle. First of all, the C’Thanks bundle was still for sale, and is currently for sale as of the time of this writing, so it is very likely that not all the people who will buy it had done so at the time the survey was taken, and they were given no option to indicate that even if they knew at that moment what decision they would make. Likewise, we don’t know how many people answered the Felosophy bundle survey indicating they were, or were not, going to purchase it but then made a different choice once it was for sale. Because these surveys are asking different things, and the methodology was sloppy, it becomes a very poor and nearly useless apples to oranges comparison. Additionally, because these two surveys were focused only on individual products, it could be that a person did not choose to buy either but still purchases other things like the battlegrounds perks or arena tickets.
About the claim that “33% of players continue to boycott Blizzard”. What does boycott mean in this context? Presumably it means abstaining from purchasing a product they would have otherwise purchased, but that is not all the word ‘boycott’ means, and void of any clarification in the responses, it introduces doubt. Boycott can also mean abstaining from playing the game. A free-to-player who was never going to purchase the bundle anyway, but who is also boycotting by not playing the game, could have chosen that as a perfectly valid answer. Or it could be that a person is boycotting the purchase but would not have purchased it anyway for different reasons not covered in the other two ‘no’ responses. You are led to believe that one third of all the players who habitually buy Hearthstone products have stopped buying things they would have otherwise purchased in order to protest the Battle Pass. That is the conclusion he wants you to draw because that is the conclusion he knows will be well received by his audience. And that brings me back to the point I made earlier: negligence or malice? In my opinion, malice is the only conclusion I can draw.
Conclusion
Having an opinion is one thing. Zeddy tells the community what it wants to hear constantly, but he doesn't dress himself up in the perceived prestige and authority of a mathematical expert. This community has actual mathematical experts working over at Vicious Syndicate, and Hearthstone Mathematics trying to glom onto same respect when he doesn’t deserve it and hasn’t earned it is worthy of rebuke.
As I said at the beginning, it is long past time to stop taking Hearthstone Mathematics seriously.
