Keywords Studios : How to work with external QA teams to get the most out of your testing

By Edgar Diaz, Functionality QA Manager, Keywords Studios
www.linkedin.com/in/edgar-díaz

So you got yourself an external QA team…

Starting a small video games studio with super-talented friends in a basement is undeniably cool.

And, in a way, it's "the dream" for both game enthusiasts looking to break into the industry, and for veterans who have spent many years in large gaming corporations and who are now looking for more control over their craft.

So, you went ahead and started a small project, and it went great.

Your studio started to grow and plans for your upcoming games became larger and more complex. Now your testing needs exceed those capable of being handled by your internal staff and you're unsure of what to do next. What can you do?

Don't worry, we've got you.

Here are a few excellent articles to help you know if external testing is right for you, as well as the critical considerations when choosing a vendor.

Let's skip ahead and assume you're at the point where you're about to start testing with an external functionality QA (FQA) team.

Having teams in two separate companies (or even countries) can be tricky and intimidating and, in these cases, clear and regular communication is paramount.

Here we list the most important things your QA vendor needs to know (or get from you) to ensure your external testing goes smoothly.

Document everything … yes, all of it!

There is no better way to understand what a video game is all about than playing it thoroughly, right?

Well, that luxury will not be afforded to your external QA vendor. When testing starts, the game will most likely be far from fully playable and it may only be a little further along than a prototype or a vertical slice.

Therefore, a game design document (GDD) is the next best thing to provide to the external QA teams.

It is understandable that your design documentation might be thrown to the side in the madness of production.

That's okay. Rarely does a GDD become so outdated that it doesn't reflect the actual game in the slightest.

So, sharing whatever design documentation you do have goes a long way, not only in establishing a common understanding of the project, but also in helping testers confirm if a certain behaviour is expected or not.

It also can save them having to go back and ask you multiple times.

Edgar Diaz (left) and team members at the expanded Keywords Studios Mexico City studio

Other types of documentation can be valuable for your test team, depending on your game's features.

For example, if you have lotteries or loot-boxes implemented, be sure to share the drop rates with the test team so they can confirm if they behave properly.

Or, for instance, if your game is a tournament fighter, let them see the balance sheets and frame counts of each move. The more visibility you give the testing team of the theory of your project, the more efficient they will become at making sure this translates into practice.

Got a technical design document (TDD) laying around? Awesome, send it over!

That way, testers can see dependencies between features and develop better test strategies. If your QA team knows that your action phase uses the same models as your character editor, whenever something appears to be wrong in one, they will know to go check on the other too.

Having technical design details can greatly empower your testers to find more bugs for you.

You may not have documented everything; you might even have no documentation at all.

That is something you need to bring up with your QA vendor, discuss, and see if together you can come up with other ways to give the team the visibility they need to understand your awesome project as well as you do.

Collaboration Tools

Prior to the project and during onboarding, think about how you want information to be delivered to you.

Most vendors will send you a daily report highlighting their findings. However, you can discuss with them what type of details you need in there, if you prefer to get that info at the beginning of your shift or at the end.

Tailoring reporting can save you time while gathering data and help you predict when you're going to be able to achieve your next deliverable, as well as when information needs to be escalated within your organisation.

The same goes for terminology and formats. Us vendors have a standard terminology, but you can always share yours. For example, your vendor might normally define a game getting stuck on the same frame and becoming unresponsive as a "Freeze", but your devs might call that a "Hang". By sharing your terminology and formats, nothing gets lost in translation.

Do you feel like you need a "Suggestion" bug category so testers share more subjective feedback? Want specific tags in bug summaries to make bug assigning easier for you? Your vendor should be happy to align these types of details to your requests, especially for longer projects.

Also, you should have a clear picture of the tools you'll want to use for this collaboration. What issue tracking database or test case management suite do you have at your disposal?

We've seen projects use the most sophisticated, dedicated software and we've seen entire projects handle their bug databases in spreadsheets. It will depend on your preferences, but always feel free to ask your vendor if they have something already in place that you can use.

Often it's as simple as creating some extra accounts and they'll host your base for a limited time.

When both parties are aligned on this, it becomes much easier to integrate bug reports into your workflow.

Likewise, build distribution is crucial not only for agility , but also for security. Most major platforms have beta environments you can use to manage who gets access to what build: Apple's TestFlight, Steam's Beta and so on.

It's convenient to use those.

In other cases, you won't have distribution methods like these available, or you may choose to use something else.

Aspera, VPN/FTPs tunnels and file sharing platforms are always an option for standalone builds. However, it is important to discuss this with your vendor early, as establishing a build distribution workflow that is secure, reliable and fast at transfers often involves more areas (such as IT and InfoSec), and it may take a few days to set up.

Investing time and effort on this will have a positive impact on the speed at which your test team can move from one version to the next.

What about immediate communication? Slack, Skype, Teams, Discord?

Do you want only a couple of people in constant communication and have them redistribute information across their teams or would you like all your dev team communicating directly with each tester?

Each of these have pros and cons and it's always a good idea to discuss this with your vendor to build the plan that best suits your project.

Roadmap

By the time you need QA, chances are you have a pretty clear picture on what your project schedule looks like; milestones, deadlines, resources etc.

You don't need to let your QA team look at every minor detail of your plan but arming them with a general roadmap and a progress status on the implementation of each feature, their priorities, and whatever blockers your team might bump into, will greatly help the testing to be well targeted to the areas of the game where it's needed, when it's needed.

Feeling a bit iffy on a certain feature because it was rushed and probably implemented by one of your more junior developers? Voice your concerns to the QA vendor!

Testers will be all over it, digging out those nasty bugs for you. Conversely, as certain aspects of the game mature, it might be a waste of resources to test them repeatedly, at least at the same level of intensity.

The closer communication you have with your vendor on planning as the project progresses, the more precise the testing efforts will become, and ultimately, the better value you'll get from your test team.

Debug features

As you know, implementing items such as cheats, save file editors, and debug info displays greatly helps your dev team to move through the game at a faster pace.

Similarly, these features can help the QA team test faster, and see more of the game in less time.

You might worry that debug features themselves might trigger bugs that your end users, who don't have access to these features, will never encounter, rendering those issues invalid and a bad use of your time.

However, it's best to discuss and strategize in conjunction with your QA vendor to agree on how to use debug features responsibly so that the team gets all of those benefits, without risking false-positives.

You can tell the testing teams exactly how these tools should be used - when, and where in the game.

It's also important to note that, in this day and age, your game is probably not an executable file that you can put on a device and run. It may have online multiplayer, DRM, in-app purchases, DLC, and/or other online features.

You need to make sure you have a safe environment for testing, without affecting live users.

Consider giving your test team access to this type of environment or, even better, provide the tools to generate test accounts, dummy accounts, reset progress, simulate purchases or edit save files.

Basically, give them whatever you think will help the team have full access to all features, as well as the ability to simulate situations that may require more players than you have testers.

We've seen clients develop full-blown debug suites to control every aspect of their game (sometimes even game servers) from outside the blackbox, and we've seen projects with zero debug features.

Both (and anything in-between) are valid approaches but, again, communication is key here and the expectations of the testing round need to be aligned to the ability you provide the team to navigate your game, and at what speed.

The more tools you provide, the faster your testers will be able to get to specific parts of the game, which in turn allows them to test more deeply.

Conclusion: It boils down to communication

Many of the things mentioned above may seem obvious to you, and they probably are.

It may also appear like a ton of prep work is necessary just to delegate a part of your production process. Sure, you can always hire a team, send them a build, and let them run with that, and you will get bug reports regardless.

However, considering how crucial QA is to achieving your goals, and how convenient it is to do it with an external QA team, it is wise to put the time and effort into clearly defining what works for you, as two projects are never the same.

It all boils down to communication.

It may take a few meetings with your vendor to nail down all of the details and iron out the kinks but, in the end, it will make your investment in external QA services much more fruitful.

I guarantee it.

Discover more about how Keywords Functionality QA services can deliver proven, quality results for your game development project.

See all news