Greedy Goblin

Thursday, November 20, 2014

Lack of failsafes in game design

I've been playing Warlords of Draenor recently (EVE ISK comes in the background anyway), and I like it:
  • Not even the leveling content is trivial. It's not hard, of course, but it's no longer "press any key" and loot drops. Yes, the "you are the greatest hero for killing that 5 sporebats" is still annoying.
  • Group play is supported. You no longer fight over eggs or whatever items on the floor with people in the area, so you can team up. I haven't played together with my girlfriend for two years, so it's a big plus.
  • I really like the bag sorting option.
  • The garrison follower missions are a great gameplay and gearing addition. Thinking and planning is rewarded. Important advice: build both lumber mill and trading post or you run out of resources. You can replace the lumber mill later.
However these garrisons became a major source of annoyance of millions of players who got stuck, disconnected, watched the 4-hours long maintenance sessions or other way couldn't play the game. The problem is somewhere in the garrison/instance code that makes players who try to enter lost or the whole server dead.

Now, I'm not blaming Blizzard for being unable to deliver a bug-free code at launch. This is a huge code. Also it's probably not a straight bug "do X, crash happens", but a "do X, slight performance decrease happens" multiplied by insane amount of players. On the other hand I do blame Blizzard for not having any failsafes to prevent a probably minor bug turning into a complete service outage.

What do I mean? They went in with the "garrison code is great or we go down" approach instead of the more sensible "whatever can break will break" thinking that put two independent break fluid circles plus handbrake into the cars to make the car stoppable in case of break malfunction. The proper action would be having a method to circumvent the whole garrison code in case of malfunction. I mean a server-side switch that stops players from using garrisons but allow questing and dungeoning instead of nothing. But the best is yet to come:

This is a command table, the most important element of your garrison, allowing you to manage followers. But it's not in the Garrison, it's in the inn (called "Trading Post" for some reason) in Telaar, Nagrand. It's not instanced, it's usually surrounded by players. So the code to place garrison elements outside of the garrison was there. All they had to do to prevent the disasters is placing them in every zones, allowing the players to access them without entering their garrison, removing the server load it creates and enable limited "garrison gameplay" when garrisons are inaccessible/turned off. Also, as a final - and not to hard - emergency feature could save lot of players: a "teleport to Stormwind/Orgrimmar button" on the login screen to solve "loading screen stuck" problems.

I hope that game developers learn from this mess and write their games fail resistant, preventing one buggy game feature bringing the whole thing down.


Anonymous said...

The command table can be accessed from any zone you have built an outpost in, so basically any WoD zone you have done about 5 quests in.

Chris K. said...

+1 on the "teleport to capital" button on the login screen, even if it's just a temp thing. Probably 90% of their CS tickets would go away overnight.

Anonymous said...

From a developers point of view:

What you are suggesting is not always possible, as such bugs are unforseen, and building failsafe only works if you are actually able to catch the problem.

Also keep in mind that failsafes like you are suggesting raise the complexity of the code and by that the actual probability of errors to be made.

Anonymous said...

"I hope that game developers learn from this mess and write their games fail resistant, preventing one buggy game feature bringing the whole thing down."

they don't
D3 was a thing. same company. launch was a disaster.
with WoD I thought "why did they bother with a so called 'beta test' at all?"

like everything software. if it compiles (to say it barley runs) lets ship it. you fail-safe concept is wishful thinking. Good honest software does sometimes have noble and good concepts to prevent stuff. but in the end if you can get away with shipping garbage, you will.

99smite said...

Nice article, althoug about a game I'll never play.

Similar things could be said about just any software company.
The trouble is that the more complex a code gets, the more difficult it gets at monitoring exceptional behaviour.

Basically you need two types of software engineers: those that want everything to finction and who think straight.
and those who are devious little devils who think outside the box and try to break a program in many creative ways. they can give the feedback needed for the former group...

Camo said...

Yes the command table is available in every zone. The main problem was that you need to build the garrison as an intro and requiring your own instance for that.
What didn't help was that people would not know (or care) about the table and rather hearth back to the garrison.
I got to level cap on launch day and once we rushed past the first quests, there was no problem as you just go to the next zone instead of going back and taking the flight from your garrison.
However anticipating that the garrison intro would be a bottleneck shouldn't be that hard.

maxim said...

Garrison is essentially player housing with a twist.

Blizzard tried to make player housing a central part of game experience (which works great as long as tech issues don't interfere). Which is why every time player housing is down, you can't advance.

Making all features of player housing accessible in their entirety without walking into the actual house kind of defeats the purpose.

dobablo said...

As noted above. There is a similar table in every zone with an outpost. There needs to be one in Ashran too.

The main bug is due to server workload issue being higher than expected. It occurs when the server reaches the personal instance cap. You notice it recurring on other quests that require personal instance.
My suspicion is that when they extended the shard splitting technology beyond the first zone (to keep hundreds of people gathering together and creating lag), they lost some of their capacity to create personal instances.

Péter Zoltán said...

As far as I understand garrisons are instanced, therefore we have more people in the instance servers than ever before.
Passing the garrison border transfers your char between the world and instance server (without a loading screen) which is apparently not an easy process either. I guess it is similar to the MoP farm, but apparently much more people try to use it at once.

They looped most of the early questlines through some garrison NPCs which is completely dumb as you cannot progress without using the garrison.

Indeed the failsafes seem to be completely missing. I still had major issues getting into my garrison yesterday - 6 days after launch.

Gevlon said...

There is a trivial failsafe for too few instances: instance queue. When you want to enter the garrison, or other instance, you get an UI element "too many players want to use instances, you are #1524 in the queue, please wait or press cancel to be teleported to the nearest village"

Anonymous said...

"There is a trivial failsafe for too few instances: instance queue. When you want to enter the garrison, or other instance, you get an UI element "too many players want to use instances, you are #1524 in the queue, please wait or press cancel to be teleported to the nearest village""
Then you'd just have the thousands of cries of "Blizzards servers suck they can't let everyone do instances together, waah".

At the end of the day, people are going to complain every time any bug appears in any piece of software, which is every piece of software in the world (since there's no such thing as bug a bug free program). As long as the company acknowledges and works on a solution, then fair enough.

Kristophr said...

The Outpost Command tables were not available to starting characters.

As Dobablo noted, they should have placed a command table in Ashran, and IMO, a plans table, and architectural assistants there as well.

Not allowing the Garrison to be built until players were at level 92, and maybe having your followers show up when you click the garrison flag at any level may have prevented the congestion.

( unfortunately, Blizz needs to allow new buyers to immediately access the new content promised to keep newbs interested, so that idea was probably a non-starter. )

Anonymous said...

You like the missions? i thought of them like farmville to the max.
that being said, They are OK.
You should consider having an Inn thoug, its imperative in getting a few good followers (epic mount!) The barn is also slightly better than the mill or so im told, but then it requires greater input so its not like its surprising.