Tuesday, January 24, 2012

Lets Talk Numbers: Raid Kills, Difficulty, and Proper Interpretation.

Ok, this has been something that I've been noticing in the recent storm of blog posts and comments about the Dragon Soul nerfs. It's been mostly happening in comments, but some of the more prominent bloggers have fallen into this trap as well. They don't know how to look at the raw data we've been given by sites like Wowprogress and MMO-champion, and turn that data into a usable analysis. From Matt Rossi:
Even if you just consider the 800,000 players who finished Firelands, only a quarter of them are done with normal Dragon Soul. This means when players make comments like "Dragon Soul is easier than Firelands," they're not at all supported by the statistics. As many people had completed Firelands pre-nerf as have now completed Dragon Soul. Pre-nerf Firelands was, statistically speaking, on par with and not harder than Dragon Soul is right now.
That's not only incorrect from an analytic perspective, it's also factually incorrect. Wowprogress shows us that there were 19,500 Deathwing normal kills by January 19th, the date that the article was published. That was 52 days after the instance was opened up on November 29th. On August 18th, 52 days after Firelands was released, there had still only been 9,500 Ragnaros kills. 10,000 more guilds have killed Deathwing than killed Ragnaros in the same time span, a 105% increase. This is an indication of Dragon Soul being significantly easier than Firelands was, not "on par", as Mr. Rossi claims.

I've spoken before about how raid groups will settle onto their appropriate place on the curve of raid kills. There's essentially three factors that decide if you're going to down a boss. There's skill, gear, and commitment. Skill is the combined ability of the members of your raid to know how to play their class and role, and their ability to learn and adapt to the mechanics of the encounter. Gear is just that, the quality of the gear that your raid has. This provides a buffer, stronger tanks are less likely to die, bigger healing throughput keeps the raid up, and more damage output shortens encounters. More gear means an easier encounter, which means less skill is required. Then there is commitment. All other things being equal, a guild that raids five days a week will progress further than a guild that raids five hours a week like Legacy. Skill x Gear x Commitment = the timeline upon which a group can expect to down bosses, assuming the difficulty is equivalent.

Many people measure the objective difficulty of encounters via commitment required to down the boss, either in number of pulls, or number of weeks spent learning a particular encounter. The reason for this is that within a raid group, skill and gear tend to be equivalent at equivalent points of progression in different tiers. A guild in 346 gear working on Halfus will take about the same amount of time as a guild in 359 gear working on Shannox. It's a rare occasion where Mr. McEatstheFloor suddenly becomes an amazing player. These things rarely change.

A guild that requires three months of farming gear in order to raise their SkillxGearxCommitment quotient high enough to down the final boss in the instance does not magically become skilled enough to clear the next instance on the first day, or in the first week, or in the first month. If you were not in the first 10K Ragnaros kills in T12, then you shouldn't expect to be done with normal mode Dragon Soul yet. However, 10k players beyond reasonable expectation have already cleared the instance. We can use this to establish just how much easier Dragon Soul is than Firelands, and through that assessment, we can build reasonable expectations for when a raid group that cleared Firelands should expect to be able to down Deathwing.

The 19,500th Ragnaros kill didn't occur until September 26th, 91 days after release. There's a 39 day disparity between hitting that benchmark in Dragon Soul, and hitting it in Firelands. You can either make the assumption that the lack of difficulty simply frontloaded guilds, that the guilds that downed Rag in the first month downed him the first week, and then things settled. You can also look at it as a compressed schedule, where every day spent in Firelands is equal to 60% of a day in Dragon Soul. It's more likely a combination of the two. However, this shows that it's reasonable to expect your group to reach a milestone in DS about three to five weeks earlier than it took them to hit that level in Firelands. It is not reasonable to expect kills to be rolling in three to five months sooner.


  1. I don't think you can discount the effect of the Raidfinder.

    I was 0/7 in Firelands. Why? The apparent unPuGability of the encounters which resulted in poor confidence and very high PuG demands for gear and evidence of previous kills.

    Of course I can't speak for all casual raiders, but Firelands was a disaster for me in getting into a successful PuG, and surely I wasn't alone. And trying to make sense of those boss strats from videos or guides was quite intimidating. The 2 or 3 PuGs that I got to see Shannox were complete clusters.

    Enter the Raidfinder, which allowed me to successfully see my first expansion end boss die. Then practice weekly, develop confidence, and eventually get in a great PuG that went 4/8. Amazingly, preparing for a boss by reading strats is much easier when you have seen even a simpler version of the fight.

    1. Reading over, I probably shouldn't even call myself a "casual raider", more of a raid tourist, but a capable one if given a chance to practice.

      I just think that the Raidfinder may be partly responsible for the increase in number of kills.

    2. You know, for some reason,I never noticed the reply functionality on my blog before. Oh well.

      In response to your speculations, I don't think that the inclusion of the raid finder had nearly as big an impact as the overall tuning of the instance. In the first week of the patch, when the last four bosses weren't available in raid finder difficulty, there were over 5,000 Madness of Deathwing kills, there were only 1,200 guilds clearing Firelands week one.

      The instance as a whole is very undertuned, particularly the first four bosses. 88.46% of groups clear the first four bosses in Dragon Soul, compared to 88.24% clearing four bosses in Firelands after all the nerfs and the influx of 397 and 384 gear, or the 87.35% of groups that managed to clear four bosses in T11 normal, after the brutal nerfs and two tiers of gear to contribute to the buffer. People can't outgear Dragon Soul, there's no higher tier at this point, and yet they've been more successful since day one than any other instance prior to ToC. It's not an experience issue, it's a tuning issue.

  2. This blog is hard to read. The black on blue makes it so. The red links are worse. I basically copy pasted the text into an editor to get through the post.

    Anyway, all numbers indicate that players either don't enjoy raiding as much as the devs want them to, or that raiding is sufficiently difficult to keep players from either trying it. I don't know which it is, but it's the only conclusion that's been true for years. Few raid and even fewer succeed at it. I personally think most players just dont find it enjoyable. I'm not one of them.

  3. The "undertuning" is a result of Blizzard's ongoing work with making raids even more accessible. For that to happen, normal modes need to be easier than they were in Firelands, and casual/regular guilds can not be stuck on one boss for months and months, because they simply do not have the dedication it takes to keep banging their heads against the wall. Blizzard wants people to see all their content. While they need to cater to the hardcore elitists who measure up against eachother on ranking sites etc., they also wish to make an entertaining game for those who don't devote all their spare time to reading up on encounters and preparing their class to the utmost. These players make up the vast majority, wether the hardcore like that or not.

    I used to raid hardcore. Now I raid in a very casual guild, 2 times a week. From my old raiding perspective, it's a nightmare. People don't prepare, don't bring flasks, food, elixirs, alternative trinkets, etc. But it's given me a new insight into how the more casual player thinks and feels, and how raiding is for the vast majority of players. Many are scared, both because they know that they will be evaluated by other more experienced players, and because they fear they'll do stupid things that'll be punished either by mechanics or their peers. Even because of that they don't prepare better, they just hope for the best. Some have good experiences because their raid leader has a zen place, but many have bad experiences and drop out of raiding altogether. I've spent countless hours by the dummies, trying to help them, pointing them to resources like elitist jerks etc., but even then it's an uphill struggle. Most casuals actually don't understand what's said on Elitist Jerks, because they haven't been reading those types of texts for long enough, and they don't _really_ care enough to get into it - they just try it because there's peer pressure.

    Saying "well if they can't prepare or be great players they shouldn't be raiding" isn't something Blizzard can really do anything useful with. They want even the not-so-great players to be able to see their content. They want more guild to kill Deathwing, Ragnaros, whatever the current badass is. From a statistics standpoint, the hardcore players screaming that the nerfs are destroying the game are a severe minority, and it's not easy to say wether it's advisable to listen to them from a business point of view. I'm sure there are thousands of guilds that barely know who Paragon or Blood Legion are, and don't really give a flying patootie. They like to raid and they like to play a game, but when their progress is severely hampered because a boss is simply too hard for them, Blizzard needs to tune the content down so that people can see it. And why the hardcore scream I have no idea. They routinely outgear even pretty serious raiders after a couple of weeks, and when the content is nerfed and "regulars" get access to better gear, its soon time for something new and even more glorious for the hardcore to bask in.

    If a guild needs months and months to break say 5 or 6 out of 8 heroic bosses, aren't they per their own definition not good enough? And if they, who consider themselves to be good to very good players, need that long, can't they see that the content will not be available to lesser players at all? Are they such narrow minded people that they actually have to feel better than lesser players in a computer game? Can't they be content that they made more headway than those players anyway?

    Finally I need to mention the color scheme on this blog, like Doone before me (if that is his nickname, it's very hard to read with the red on blue) - it's super hard to read. Other than that I quite enjoy it and thank you for your insight and thoughts.