Log entry GD9: Survivor Error

This week we have been covering the business aspect of games and how analyzing the playing field goes a long way in determining the road to success. One of the readings we had this week provided data and trends related to the top products in the mobile industry. If we look around there are plenty of articles and reports evaluating the top companies and their products in every field. And this is a huge thing today isn't it? Even within a game. Big Data. The power of recording every stat and more importantly, the power to analyze it in excruciating detail. But occasionally, we get bogged down by all these details. We end up looking at only the success stories. Actually, let me rephrase that: we are only allowed to look at the success stories and this leads to one of the most interesting logical fallacies: Survivorship Bias.

For those who are unfamiliar with Survivorship Bias, let me give you a military example to explain it:

During WWII, the chances of a bomber crew surviving a mission were almost 50-50. Even if they managed to make it back, the prolonged war meant they would fly out again soon until their luck ran out. It was crucial to find any kind of modification that would improve the odds of a bomber making it home. The engineers knew that the allied bombers needed more armor, but they couldn’t just cover the planes up like tanks, not if they wanted them to take off. They had to carefully choose the sections to be reinforced. They looked at the planes that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw that the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner.

The commanders wanted to add the thicker protection where they could clearly see the most damage, where the holes clustered. But Abraham Wald who was advising them said no, that would be precisely the wrong decision. The holes showed where a bomber could be shot and still survive the flight home, Wald explained. After all, here they were, holes and all. It was the planes that weren’t there that needed extra protection, and they had needed it in places that these planes had not. The holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit.

A similar logical error is possible while evaluating video games when we consider only the ones that do well. There are definitely some insights within success stories but it's equally important to consider the titles that failed or others that you probably haven't heard of before jumping to conclusions.

Let's take the MOBA genre as an example. League of Legends is probably one of the most successful games out there today and is helping bring e-sports into the limelight with a very professional approach. I have played LoL since season 1, so I have seen it evolve from its rough, unbalanced and unproven origins. League was considered to be a cartoonish clone of DotA, meant for casual audiences and most people dismissed it in anticipation for the release of DotA 2. If you look at the success story of LoL, you'll find that its revenue model was unique. More importantly, Riot stuck to it throughout the early days even when there weren't many players invested in their game. This gave them an edge over competitors like Heroes of Newerth. But there was more to the game than just its business model.

Before we get into those details, let's take a look at some of the recent MOBAs that have failed. These were games who had plenty of time to observe the successful titles, yet they still did not make the cut:

1. Sins of a Dark Age

Status: Development ceased
Reason: More of a DotA clone that LoL, SoaDA hardly innovated anywhere and had no new features to boast about. To make it worse, their UI lacked the polish that the others had and thus it couldn't generate a strong player base.

2. Dethroned!

Status: Funding/Development ceased
Reason: This was a unique game where they tried to mix MOBA and RTS elements. The visual style was simple and so was the gameplay. A little too simple actually. And that's where it lacked depth that players expect from this genre. In addition, there were lots of bugs that weren't ironed so again they couldn't retain enough players to continue development.

3. Panzar

Status: Near empty servers / close to extinction
Reason: Panzar was a game with pretty graphics riding on a strong engine. But they lost the plot with the grinding involved for gaining the gear and everyone recognized it as a pay to win game. This led to players abandoning the game over time.

4. Dawngate

Status: Development shut down
Reason: Dawngate was expected to be the next big thing especially since it had EA backing it. But that may be the reason it never took wings. The game had a decent start and seemed to be in good shape for a Beta version but the plug was pulled because it didn't attract a crowd large enough to take on LoL or DotA2.

5. Infinite Crisis

Status: Servers shut down
Reason: IC was a title that tried too much without paying sufficient attention to detail. They had a strong backbone in the form of the DC universe, but they didn't build upon that. There were several complaints of negative interactions with the developers who were also notorious for not communicating at all. The game had balance and toxicity issues that were not addressed. And instead of the refining these features they were focussed on e-sports and Twitch even though the foundation was weak.

Thus, in addition to not forcing a player to spend money, Riot made a lot of effort in different areas to stand out:

  1. They embraced their design of being a casual MOBA. They changed elements like denying creeps and losing gold on death. These simple changes made it easier for new players to get into the game without getting stomped by more experienced players. It also allowed the losing side to have comeback opportunities which would keep games interesting throughout.

  2. The developers were quick to fix any major bugs and abusers were handled well. The patches were updated regularly and there was a constant effort to balance the game.

  3. Throughout their initial years, Riot was constantly releasing champions and other content into the game to never let it get stale.

  4. The Riot staff actually spend time playing with real players to get an idea of the game and some decisions are based of these experiences. A lot of it is now decided by competitive play but they still have their ear to the ground.

  5. Riot has battled a lot with toxicity and experimented with things like the Tribunal and other incentives to address such behavior.

  6. Within e-sports, they have also made continuous effort to improve the production value and they approach the broadcast very professionally - their quality was considered high enough to be featured on ESPN.

While money eventually defines success, just utilizing a free-to-play model like LoL is not enough to create a successful MOBA. There were a lot of other factors which led to its success and these traits can only be revealed when you contrast it against the ones that stumbled after their initial steps. That's why it will take much more for big games like Smite and Heroes of the Storm to scale those heights.

There is another example much closer to home. While designing for my current project, I had to figure out the experiences that kids aged 7 - 11 years would like. My natural instinct was to look up the top rated apps on the store. It was difficult to find references for this age range, but I eventually stumbled onto a few good apps and it was easier to explore further by looking at related content. Based on this initial research as well as the observations from previous projects, it seemed like the key to an enjoyable experience was simple: 'Customization'. Just provide a ton of options for children to express themselves with. But customization was a feature that did not match our project goals. There needed to be some other hook within the experience to keep the children engaged. Thus, I started looking at mechanics.

Tap, swipe and drag were the most widely implemented gestures in any game meant for a wide audience. The few things I learned here were:

  1. Keep the interface as clean as possible. Do not add complicated gestures or controls.

  2. The tasks themselves should be discrete and easy to perform.

  3. There were some age specific tips here:

    • Avoid keeping interactive buttons at the bottom of the screen as kids are likely to press them by accident.
    • Children are able to perceive when an experience is designed for an age level older or younger than themselves and may have strong opinions regarding it.
    • 'Challenge and Reward' is the favored approach that most of these games take.

Now, if looking up the top apps was difficult, searching for the bad ones was close to impossible. There is no way to sort the apps by rating (possibly to prevent cheating) and looking at the bottom of charts meant that I couldn't filter for the required age range. Another hurdle was that I was looking for apps which were bad in terms of design and not because of lack of effort put in. There are 1 and 2 star rated games that are basically interactive memes but I was looking beyond that. I needed to check out games that had decent production value but still failed. After a lot of manual searching, I found some games which revealed other aspects of a tablet experience that I might have missed completely:

  1. One of the main reasons for player discontent (apart from micro-transactions) is inconsistent performance. Games like Breakneck, a racing game with frame-rate issues and Disney Color and Play, which had unreliable page scanning both left their customers very frustrated. These issues were enough to tarnish the reputation of experiences that had the potential to be really successful. I realized that all the top games I had researched previously had one thing in common: No matter how complex the world was, there were no bugs or performance issues. The experience was seamless.

  2. A game like Feed the Duck proved the importance of understanding the needs of your audience and the genre you are creating for. It should come as no surprise that a physics game with bugs and bad world feedback was not very successful. While the art and concept were unique, the shallowness of the game prevented it from attracting puzzle game enthusiasts.

  3. And finally, Final Fantasy: All the Braves showed me how it was necessary to deliver on the player's expectations. A player installing a FF game awaits the moment of battle when they would issue orders and watch their actions play out. Instead the combat in this game devolved into swiping the screen. That's it. Really! Go check out a gameplay video. Some of the animations are not even visible as your finger obstructs the view as you desperately try to swipe everywhere. Even big titles are not immune to such decisions.

Well, this has been a long post. My main idea was to bring attention to the fact that when you research or analyze something, it's vital to look at both ends of the spectrum to gain valuable insight. When the focus is just on the things that fared well, it's easy to overlook certain traits that we take for granted. Sometimes, these are the qualities that end up being crucial towards the end goal.

Credit to David McRaney for serving as the inspiration for this post.