Greedy Goblin

Friday, March 20, 2009

Statistical evidence

Nothing is sure in several fields including business. Does X sell for Y? Nobody knows. This encourages people to return to their favorite idea: there are no facts here, everyone can have his opinion on the topic and all opinions are equal.

They love this idea, since "being right" is very important for social people as it is connected to positive personality traits like "smart", "wise", "just". The "everyone can have opinion and all opinions are equal" means "I can be wise without thinking a bit".

Look at this comment for example:
"Everyone stands in the fire even a tiny little bit sometimes. In my opinion, at some point in their raid career, everyone takes a bit damage they probably could have avoided, if they we're playing a perfect game. So where do we draw the line between how much standing in the fire is bad? Does one fire wall or void zone, once, ever, make you instantly an M&S... clearly not... And therefore, on any boss which isn't Patchwerk (such as my Malygos) your "conquering" that raid boss is not a fact, but an opinion held by you?"

Since there are no facts, just opinions, respect all opinions, therefore all opinion holders! In short: respect me! (despite I'm doing nothing but sitting in the basement talking nonsense)

Bad news punks! The science of statistics is here to ruin your pink dollhouse. While everyone eats fire sometimes, the average amount of fire eating varies. This average can be measured if enough data is available. So let's see an oversimplified situation where the damage is not spiky at all. If the fire damages X and you eat P fires in the fight, than the healers have to heal X*P extra. If the healer have Y healing capacity, than you occupy X*P/Y portion of the healer. So because of your fire-eating, the raid has to bring X*P/Y extra healers instead of DPS. If the "good enough" DPS is Z, you have to provide Z*X*P/Y extra DPS to make up for it. Simple example: if you eat so much fire that you need a dedicated healer, you must DPS two times more than "good enough" (one for yourself, one for your healer) or you are carried.

Now let's see something much nastier, spiky damage: At Sartharion, one healer can save one fire-eater every wave. If fire-eating is very rare, the raid healers will handle it. If there is 10% chance that a person eats a flame wave, how many people eat the same wave in a 25 man raid? The probability mass function of the binomial distribution answers P^k*(1-P)^(n-k)*n!/k!/(n-k)!:
  • 1 person eating the same wave: 20%
  • 2 people eating the same wave: 27%
  • 3 people eating the same wave: 23%
  • 4 people eating the same wave: 14%
  • 5 people eating the same wave: 6.5%
  • 6 people eating the same wave: 2.4%
  • 7 people eating the same wave: 0.7%
Let's say there are 10 waves in the fight. If we bring 4 extra raid healers there is 6.5% chance to lose a player every wave and 2.4% to lose two, 0.7 to lose three. (6.5+2.4*2+0.7*3)*10=134, that's losing 1.34 player, it's better to take 5 extra raid healers. The DPS must damage for these extra healers too. If we also consider that the healers are imperfect, we need even more. So we got back to the classical "good enough" question: can the DPS damage enough for the extra healers to kill the boss before enrage?

Of course not every player are equal. Every player has his own flame eating chance and DPS (or flame saving heal chance). These numbers can be put into the equations and we can get a precise answer for the question: is this person is pulling his weight or is he carried (in a sense that 24 clones of him could or could not complete it)?

Of course it's easier to not calculate, but say "we can't calculate, we can just have opinions". No! Maybe you cannot calculate it, but it still can be calculated. Just because you don't know the numbers, they exist. And they exist on every possible field where things can be measured at least statistically.

Since in business everything is in numbers, everything can be calculated. Of course it does not mean that I surely can calculate them. It does not even mean that anyone can do it. Just because the data exist, it does not mean you (or anyone) can access it. Just because the formula to calculate the result exists, it does not mean you know it.

In such cases the right answer is "I don't know". But people hate this sentence. It sounds like "I am incompetent", or "I am dumb". So they rather not say it. The more honest ones make up the good old "no one can know it, just opinions exist", the less honest ones simply guess and claim that their opinion is "expert" therefore you shall listen to them.

Such guys gave AAA to Lehman, such guys declared low income guys loan-capable, such guys let the economy go down. And other people listened to them without asking for numbers because they are so used to "no one can know it, just opinions exist".

I did not afraid to say "I don't know". I wasn't too happy with it. But that makes this site reliable. You may don't find what you seek here, but you will find no desinformation either.

19 comments:

Beltayn said...

The goblin makes a fair point. Our society has somehow gotten to the stage where saying "I don't know" is far worse than saying "I know" and then screwing up. So much for honesty I guess.

Also, we're not allowed to blame anybody for being wrong anymore. There's always an excuse. The excuse is never, "Geez, I'm sorry guys, I really screwed up, I said I knew when I didn't because I wanted everyone to respect me." That's not the excuse you get, instead you get something like: "LOL WHO CARES?" (Have a look at the way Ivan Sulik handles stuff, he's a prime example and he's still employed: http://www.brokentoys.org/2009/03/19/cryptic-marcom-malreported-verify-ungood-rectify-candygive/)

I've got far more respect for people who don't waste my time and money by telling me they don't know. When they say they DO however know something, I'm more inclined to listen to them.

I have absolutely no problem explaining boss fights to people in WoW if they say they don't know the fight. If however they say they do know it and then wipe us with "LOL I DIDNT NO U HAD 2 STAY OUT OF THE FIER LOL!!!", well there's going to be a very quick party kick and ignore coming up.

Anonymous said...

Good article once again. Your blog is always an interesting read, even when you don't play WoW I think.

DarkKnight said...

Quoting Plastic Rat:
"I have absolutely no problem explaining boss fights to people in WoW if they say they don't know the fight. If however they say they do know it and then wipe us with "LOL I DIDNT NO U HAD 2 STAY OUT OF THE FIER LOL!!!", well there's going to be a very quick party kick and ignore coming up."
Oh God yes, words can not describe how much I agree with you there.
Also the article is quite true and shows indeed a huge problem in today's society with the not allowed "I don't know"-sentence.

But yeh, I already see it when a girlfriend and I organize funraid to 'oldskool' instances of Vanilla and TBC, where we ask people to know tactics. And each time we ask if people know tactics, of course no-one says 'no'. Still when we get to a boss we suddenly hear: "oh, did you have to do this/that". Just makes me /sigh.

Anonymous said...

A few comments to your post:

You assume fire eating to have a binomial distribution. I think that if you would really analyse the data, you will find a Pareto distribution a.k.a an 80/20 rule. 20% of the raiders taking 80% of the damage.

Many economists use gaussian or binomial distributions because they are mathematically convenient. Calculating with averages and standard deviations is much easier than looking at the real distribution, for which ofter no mathematical tools exist. Combined with the math aversion of so many people, this can easily lead to wrong decisions.

What is interesting is, that there is evidence that it is the same misunderstanding of distributions has played a major role in assigning credit ratings. Risk assessment often uses convenient statistical distributions, instead of the real distributions, and therefore systematically underestimates events with a small probability and high impact. Find further discussion on this issue in this link: [link=http://www.fooledbyrandomness.com/fortune.pdf]fortune[/link]

Back to WoW. I have to admit that I have a terrible record of standing in fires. While I often top the healing meters, but am far from the best geared healer in my guild, I sometimes have to cause myself to defocus from the healing meters, and pay more attention to situational awareness. One thing that I have found very helpful is to use audible cues for events instead of visual ones (which I use for healing).

My point in all of this: really knowing and reflecting on the mechanism that is causing a system to fail is indispensable.

Anonymous said...

I think the point of the original comment, is that if eveyrone was playing a perfect game, you can farm Naxx25 with 3 or 4 healers.

If everyone spends a little more time in the fire, you need 6 or 7 healers.

Therefore "good enough" is objective, since it depends on the standards of the group you're with.

I'm sure your formula will hold up for 90% of the guilds out there, but how many members of Ensidia eat a fire wall these days? Do you think any of them have a 20% (your example) of eating a fire wall?

Do you therefore think that you're good enough for Ensidia, just because, according to your maths, you're good enough for the encounter?

I would say any average, good "enough" player, if they went to naxx25 with 24 people from Ensidia, would be boosted - and if they went with 24 pugs, they probably arent.

Therefore, "good enough" is objective - if you want to say whether you were carried or if you pulled your own weight. Ensidia's opinion is you're a "M&S" - pugs see you as a god.

Carra said...

Statistics are so much easier to interpret with pretty graphs.

Some part of me always wants to grab a calculator or spreadsheet to prove that A>B...

Sydera said...

I have absolutely no idea what this post was really about.

I'd put in a thesis statement somewhere near the top if I were you, Gevlon :)

I'm sure there's an argument here, but I can't begin to imagine what it applies to.

Was it about raiding or business? I'm not certain.

Gx1080 said...

It says a bunch of important things:

*We can insert human mistakes in calculation and determine how much healers you need AND how much DPS you need to do for being able to kill the boss despite the extra healers.

*Not everybody is capable of doing such calculations so its safe saying "i dont know". IMHO, admiting your lack of knowledge is the first step to learning (dont remember where i read it).

*People prefer saying BS that admiting that they dont know, and they are praised for it./epicfail.

PS: I cant believe that Ivan Sulik is still employed and yes ive started blogging (little self whoring).

Anonymous said...

HAHAHAHA I stopped reading after "pink dollhouse"- you owe me a new keyboard.

Anonymous said...

A fantastic point Gev and I applaud you for having the courage to say "I don't know". As you correctly pointed out, if more people in the financial indsutry had the strenght to utter such a phrase perhaps we wouldn't be in the current RL financial mess we're in. Reminds me of an old Mark Twain quote "There are lies, damned lies, and statistics."

At WoWenomics we made some predictions as to market movements of WoW items (thanks for the link by the way) and we will be writing a follow up article after 3.1 hits to show what we got right and what we got wrong. The intent being to show both that you can't get everything right and what our accuracy rate was. This is also called transparency and if all people in banking and finance believed in it the world would be a better place.

Which reminds me of another quote often attributed to Twain, "A banker is a fellow who lends you his umbrella when the sun is shining and wants it back the minute it begins to rain."

Kudos again for having the balls to tell it like it is!

Neil said...

Statistics are a powerful tool for measuring very large sets of data, but for measuring the performance of an individual over the course of a five-minute fight? I think the chance of an anomaly would overwhelm the merit of the statistical analysis.

In other words, for the purpose of WoW, I don't think there is a difference between "it cannot be calculated" and "it could only be calculated with a team of PhD's and a supercomputer".

I think the best bet for gauging things like how many healers are needed is simply to use a Monte Carlo method ... try a fight many times and build a model based on how it goes with various inputs.

Ceolwulf said...

Good points, well made.

Yaggle said...

I respect "I don't know" because a lot of times it's the most honest answer. People like to be right so much....but it's better to be honest when you don't know. Is there a God? I don't know!

Soulman said...

@ Cingy

I use shamanfriend to play a dong when my water shield falls off, or my earthshield falls off it's target. Has helped a ton. Also dbm will play loud noises at you then boss mechanics happen aka polarity shift. WoW can only talk to you through 2 sense, sight and sound and usually sight is being overwhelmed. I'm hoping someday someone will make a mouse that vibrates and i can program it to shake when something happens.

Anonymous said...

"Statistics are a powerful tool for measuring very large sets of data, but for measuring the performance of an individual over the course of a five-minute fight? I think the chance of an anomaly would overwhelm the merit of the statistical analysis."

Measuring a player or raid over the course of a single boss attempt is too small a data sample.

Measuring the group over a series of attempts begins to give you a reasonable sample size. Measuring all the people in all those attempts certainly gives you more data to help in this regard.

I cannot know if player X is a problem for my raid if he dies one time in an encounter. After raiding many weeks with him I do have enough data to know his strengths/weaknesses and I can begin to quantify those and compare them against his peers.

You only need thirty data points to determine if a set of data is "normal".

Anonymous said...

One thing that I have found very helpful is to use audible cues for events instead of visual ones

Is there an addon for that? I frequently hop out of "the fire" (or whatever) after sending a heal to "not-the-tank" and then realizing "hey! that's me!". Which beats the hell out of not jumping out of the fire, but I'm working on avoiding it before it does enough damage to trigger my "oh, he needs healing bad!" reaction.

Back to the main topic now :-)

I seldom say "I don't know", but I will label guesses as guesses. "Not sure, I would guess you need 3 healers", or "I'm guessing people who sell titansteel cooldowns lose about 10 gold per bar vs. buying the mats and AHing the bar" (that second one isn't that hard to compute, I have just been too lazy to check the price recently)

Anonymous said...

1.) Statistics are far from unbiased, and really have a very minimal baring on what the truth behind them is. Take you average for example. There are 3 different ways to calculate and average. Each way will give you a very different number depending on the deviation in your samples.

2.) Cingy said everything that needs to be said in response to Gev's post.

3.) ok maybe he missed one thing that someone else got.

"Therefore, "good enough" is objective - if you want to say whether you were carried or if you pulled your own weight. Ensidia's opinion is you're a "M&S" - pugs see you as a god."

Gev.. your blog is very entertaining so I give you kudos for that. However, it also shows your close mindedness and your ability to justify, justify, justify your own opinions even when they are contradictory to each other.

Gevlon said...

@Cingy: Obviously in a certain raid a few people makes most of the trouble. However to write the "good enough" line, we must assume equal people, since the definition of "good enough" is "could you do it with 24 equal people". Being good enough does not mean that you will down the boss, it only means that you pulled your weight. You can still wipe because others do not. In such case you have to analyze the actual raid's data and find the underperformers.

@Anonymous: I think you mean "Therefore good enough is SUBjective, since it depends on the standards of the group you're with.". I disagree. It is an objective limit. You COULD down the boss if the others would also pull their weight. To down the boss you don't need to be better player but should find better group. However if you are not good enough, you can't find a better group since it would not take you, therefore you must increase your skills.

@Sydera: the post was mostly about: There is ALWAYS an objective standard and you can NEVER hide behind "oh I have different opinion about the standard". You may only say: "I don't know the standard and my guess is...". And there were smaller points listed by Phoenixboy

@Neil: about a single fight we can only determine who did and who did not performed good enough this fight. Of course there are luck factors here. However it changes the meaningless "please focus more ppl!" into "X and Y please focus more of be kicked"

Anonymous said...

great post, and an interesting to ponder. one point of grammar police "I did not afraid to say" in last paragraph would be clearer as "I am not afraid to say" or "I was not afraid to say" depending on tense. The current incarnation while possibly grammatically correct, I will have to check, is not proper English as I have known it, and is somewhat jarring.

Tego
casual sometimes hardcore, sometimes M&S, sometimes God like protadin