ENDAVA PLC
Testing | Sebastian Denzer |
19 October 2021
An essential aspect of software development is assuring the quality of the code one has developed. Testing is an important feedback element for developers, but testing a larger piece of software can be very time-consuming. As developers, we are basically looking for the most efficient approaches - and when it comes to testing, automation helps!
WHY TESTING IS IMPORTANT
The consequences of poorly tested software range from blemishes and malfunctions to a non-operable or non-functioning application. And users are merciless in their feedback on poorly tested software - and rightly so!
If software is already in live operation and you add even a small new feature, you still have to ensure that the entire application functions smoothly and as expected after the update. That means you need to test your app - again and again. Manual and automated testing are both common software testing mechanisms.
The traditional approach is to have everything tested by the QA team, who are manually testing each part of the application. They will not only find bugs but also expose other problems that automated testing cannot cover, like bad or misleading UX, missing dialogue or localisation issues. The downside of manual testing is that it takes individual testers hours and hours of work.
In many cases, automated testing can be a very efficient alternative and make your developers' lives a lot easier - be it API testing or integration testing, where you write code to verify the functionality of certain functions, code parts, or entire software modules.
We have seen how essential testing is in general. Of course, it becomes even more important for a live product such as the multiplayer online board game Catan Universe, which we developed for our client United Soft Media GmbH in collaboration with Catan GmbH.
The "Universe" part of the name indicates a whole new level of complexity: the digital version of a whole universe of game variants and modes, including two games in one! Both come with different scenarios or maps, combinations of different game rules, settings for each game, etc.
Two games in one: Catan - a board game for up to 6 players (left), Rivals for Catan - a card game for 2 players (right).
And on top of that, Catan Universe offers both a single-player and a multiplayer experience, with the latter one using two different matchmaking algorithms to connect multiple clients, i.e. "players". The multiplayer part itself currently features games for 2-6 players. Behind the scenes, we are also looking into ways to increase this number even further. One day we might be able to allow 100 (in words: one hundred) and more players to play one game of Catan simultaneously.
OUR SOLUTION: AUTOMATED PLAY TESTING
Catan Universe is a long-running product, being first released in 2016 and enjoying continuous growth over the years. The overall variety of different game setups with all the different game situations - especially in the "endgame" phase which can last up to 1 hour - could not be covered manually by even the largest QA team. We needed a method to test all game permutations with their different modes and scenarios automatically. So, we built something that we call "Automated Play Testing".
What helped us from the start is that we already had an AI (Artificial Intelligence) implementation for both single-player and multiplayer games: in the game, you can play against well-known Catan characters like Marianne or Louis who bring in their own distinct personalities and game approaches. And while the players enjoy competing against these characters, we also use them as our very own QA testers who can play the game the whole night - without needing pizza and caffeine. Of course, they can never be as clever or creative as human players, but they can cover each and every game situation, and they can do it repeatedly without ever getting bored!
Having our testers in place, what else do we need?
First, the game must start in some kind of "autopilot mode" in which it will simply log into a user account based on a config file containing the autopilot settings, e.g. described as JSON data.
{
"autoPilot": true,
"email": "autopilot@automail.com",
"password": "123456",
"autoPlay": false,
"turboMode": false
}
Secondly, we want to enter the autopilot mode only in certain situations, so we use command line parameters to control which special functions should be enabled. For example, we differentiate between "active" and "passive" game instances where one active player will invite other, passive players to a game.
CatanUniverse.exe /autopilot /config user1.json
Once we can automatically log in, we should start giving the autopilot input on what it must react to or even start its own logic to communicate with other clients. These could be
- friend or guild membership requests to build a player base we can interact with,
- lobby invites for "custom matches" (inviting other players that we have in our friend / guild list),
- or the game client could start the "auto-match" matchmaking (searching for any players that match your favourite game settings).
Autopilot mode: automatically logging in and starting a single-player game - without any user interaction.
We keep the autopilot code separate from our game logic code, so that none of our developers can accidentally break the game logic. Of course, we also want to strip all autopilot-related code from the final release build of the game since we do not want players to use the autopilot AI - they should play on their own! And finally, we definitely do not want to open any security issues by allowing automatic control of multiple game instances - think of DDOS (Distributed Denial of Service) attacks, for example.
Which brings us to the final puzzle piece: to be able to start as many game clients as we want, we need some external tool to allow so-called "multi-boxing". This is a technique that some players of online games use to play with multiple characters simultaneously, e.g. for getting an increased amount of resources or experience - or only as an "additional challenge", like playing chess against multiple opponents at once.
Multi-boxing software allows us to start an application multiple times on one single computer, giving each running instance a separate set of files to work with - in our case the user config file - so that they do not interfere with each other.
Now that we have the basic pipeline running with multiple AI players testing the game, we can consider some more advanced settings: it could help to implement a "turbo mode". As the video below shows, the game is running at an incredibly fast speed. In addition, this also functions as stress test for the servers' multiplayer session handling because the AI will send game actions to the server faster than any human being could ever play the game.
You may also want to add logging and tracking systems, so that you don't have to visually check the hundreds of instances running. Actually, this system should run day and night in order to cover all game modes and situations, find any crashes or blockers and even identify situations where the AI stops playing the game because it could not find any valuable actions to perform, thus helping to improve the overall game experience.
Finally, that also means that you should integrate Automated Play Testing into your CI pipeline, so that it runs like a regular automated test on every new release you are planning.
We developed our Automated Play Testing solution in order to increase the quality and lower the costs for the extensive manual testing efforts required to validate the application. Although the initial implementation was cost-intensive, and the maintenance and adding of new features moderately increases the cost, the solution is more economical than the manual effort that would have been required for testing as well as ensuring and improving the quality of the product.
Attachments
- Original document
- Permalink
Disclaimer
Endava plc published this content on 19 October 2021 and is solely responsible for the information contained therein. Distributed by Public, unedited and unaltered, on 19 October 2021 08:51:02 UTC.
