Video Game Testing

MC Etcher here!

To start out, I thought I’d answer the video-game industry questions Paulius tossed out in the Casting Call post.

(From 2000 to 2006 I worked in various roles in Video Game Testing – what we call Quality Assurance in the business).

~~~~~

What’s it like to test a game?

For the first couple of hours on a new game, it’s fun – sometimes even less time than that, depending on the game.

If you’re testing Soldier of Fortune for the Dreamcast, using a controller for an FPS gets old real quick – but hey, it’s still an FPS. Then, when you point out to the manager that the Dreamcast keyboard and mouse just came out today, you get to have a field trip to Best Buy to buy 10 sets, come back and half the team gets to test that way! Yow!

Or, you could be testing Barbie Horse Adventures, or Pre-school Edu-tainment software, or Polly Pocket – Super Splash Island for the GBA. For weeks. And while emulators exist that allow you to play GBA games on a TV - your company only has 2 of them, so you and the other guys on the team will be squinting at a 240x160 screen for 10 hours a day for a few weeks.
.

Testing games is not playing games. It’s work. You have to test and retest every menu, every setting, and every tool in every corner of the game. You hop around in every niche, you tap on every wall, you prod every prop, frob every object, every single powerup and doodad. You have to grope everything, basically. 1000 times.

…Interesting – the green powerups can’t be collected when Polly is wearing blue bonnet with the white flower, but can be collected while wearing the blue bonnet with the yellow flower. And if you change bonnets while collecting a yellow powerup, you’re teleported back to the beginning of level 1. And your save files are corrupted.

You have to be able to find that bug, that tiny annoying bug where if you have the music and sound set to OFF and the difficulty set to GEE GOLLY, the candy launcher makes a POP sound, but only when shooting at Super-Cute-Bunnies at the bottom level of Polly’s Waterpark.

But you can’t just type in the coordinates and teleport to that position to test the next version, to see if the bug is fixed.

Oh no my friends.

The crash bug only happens during a full playthrough. Which takes 9 hours on GEE GOLLY difficulty. Which is difficult, yes - even for a game tester. And there’s no sound or music at all, since that’s what you’re testing for. So you’re bored to tears by how quiet it is, and no you can’t listen to your iPod, you’re supposed to be listening for that POP sound.

Now, just that POP sound by itself would only be a C issue (minor-the game is still shippable) but it turns into a more severe issue when you get to the Bunny Boss at the end of the last level, when the POP turns into an overlapping cacophony of POPOPOPOPOPOPs that overloads the audio buffer and causes the game to crash – but only when using the Strawberry Gumdrops.

But you need to use the Strawberry Gumdrops to blow through the doors on the boss level – so just to reach the boss, you need at least 9 Strawberry Gumdrops. There are only 11 Strawberry Gumdrops in the level, and it takes 3 to kill the boss. So you have to start the level with at least 1 Strawberry Gumdrop in order to be able to test and make sure the crash bug is fixed in the new version of the game. But it takes 8 hours to get there.

Just use cheats to give yourself unlimited health and unlimited ammo – good to go, easy to test now, right?

Wrong. With the cheats, the bug doesn’t occur. You have to test clean, just as a user might. And the cheat has caused a different crash issue, this one on levels 3 and 8 - so we had to remove the cheats completely.

And level 4 – the coolest one of all, (the only good one) had to be removed since we lost the rights to use the background art (The Smithsonian is patented? Who knew?) and the artists are in Korea, and they’re off all this week for a religious holiday. No, no one else on the planet could possibly re-do the art in time - even though it's just a wrapped/repeating jpeg about 30 pixels high and 100 pixels wide that you could make yourself with MS Paint in about 2 minutes. Nope, gotta lose level 4.

So you get to play through all 27 levels of Polly Pocket – Super Splash Island on GEE GOLLY difficulty, with no sound or music, no cheats, 197 times in 3 weeks - each attempt requiring 9 hours, and the last 15 times you do it, it’s without sleep, because this game is supposed to ship tomorrow, and the company has paid WalMart $15,000 dollars PER STORE for shelf space for Polly Pocket – Super Slash Island – right at 8 year old kid eye level – starting in precisely two weeks. If we don’t have our game on that shelf, they get to keep all the shelf money space and put something else there (getting paid twice for shelf space).

So you’re playing Polly Fucking Pocket at 3:47AM on a Tuesday, it’s your birthday, Spring Break, and your Wedding Anniversary, and the stupid Bunny Fiend Mini-boss kills you before you get to the last door in the Boss Level. Which means that it’s going to take at least 9 more hours to determine whether or not this crash bug is fixed or not.

You tell your QA Manager that you have to start over. He tells you to go ahead and start over – he’s in charge of four other games, 2 more of which are also running late. He calls the Producer, who spent a lot of time hanging out trying to be helpful and buying you all dinner from Tony Roma’s – steak dinner in Styrofoam is bizarre – but who is now at home, asleep.

The producer says “Fuck that bug. Ship it.”

And so, a few weeks later - some poor schlub who is playing through his kid sister’s Polly Pocket game on GEE GOLLY difficulty, who turns off the sound and music because they are too damn cutesy gets alllmmmossst to the boss, when the game crashes. He hurls the GameBoy against the wall, breaking it into five pieces.

~~~~~

Have you ever actually said "Can you believe I get paid for playing games?"

Yes. But the long, thankless hours and day after day of the same BS leech the fun out of it.

~~~~~

How come games can be so thoroughly tested, but still released almost "unfinished"?

There are at least 4 different reasons for this.

1) The Marketing Department schedules the release dates. This means that a game can have a completely unrealistic development schedule, but it still needs to be in stores by December 1st.

2) We don’t have the budget for or the time right now for the programmers to learn that new, cutting-edge game engine that just debuted. Sega just put out Rainbow Brite, Starshine Goddess using the OMFG02 engine, but we’re still using the UGH52 engine. This means that Polly Pocket – Super Splash Island won’t be as graphically rich, dynamic, or responsive as Rainbow Brite, but we have to put out something to compete with Rainbow Brite, or they’ll dominate the GameBoy Girls market this Christmas.

3) Because there’s such a rush on the project (for an app that is clearly outdated before it’s even released) the game designers, artists, programmers and testers have about half the time they normally do for production.

4) To meet the shelf date, the game is pushed out early, with known issues. Don’t confuse 'thorough testing' with 'thorough fixing'. I’ve worked on games where we found and wrote up thousands of bugs, but the game still ships with 800 open issues (Open=still occur) because there just isn't enough time to fix them all - and inevitably, fixes cause new issues. Most of these are minor, and while noticeable flaws, they aren’t crashes. A few of the bugs in the shipped out product are major, but typically rare – like the Polly Pocket crash bug I talked about.

Any other questions?