When testing software, testers need to be able to identify all possible scenarios that occur in the real world. The only way to do so is through in-the-wild testing.  The following is a guest post is about “The Kobayski Maru Test”, and the importance of real world simulation in testing. The post is written by David O’Dowd, a Agile Testing Consultant with BearingPoint, Ireland. David has been working with BearingPoint providing consultancy and helping a wide variety of clients to efficiently plan agile testing activities critical to sustaining and delivering high quality IT Services. With 6 years’ experience in the IT industry, David is an agile testing evangelist specializing in Agile Testing/Agile Test Management.

Cadets on the command track at the Star Fleet Academy must perform The Kobayshi Maru test before being allowed to command a ship. The test involves the cadet being put in a replica starship bridge and given the role of Captain for the duration of the evaluation. In the simulation the candidate must command a crew operating in the other key roles of the ship while facing a challenge put forward by the Academy. Think of it like a flight simulator on steroids. The challenge is as follows:

“The year is 2280, the cadet receives a distress signal stating that the Kobayashi Maru ( a civilian fuel-carrier ship holding 81 crew and 300 passengers ) has struck a “gravitic mine” in the Klingon Neutral Zone and is rapidly losing power, hull integrity and life support. There are no other vessels nearby. The cadet is faced with a command decision:

Attempt to rescue the Kobayashi Maru’s crew and passengers, which involves violating the Neutral Zone and potentially provoking the Klingons into hostile action or an all-out war;

OR

Abandon the Kobayashi Maru, potentially preventing war but leaving the crew and passengers to die.

If the cadet chooses to save the Kobayashi Maru, the scenario progresses quickly. The bridge officers notify the cadet that they are in violation of the treaty. As the starship enters the Neutral Zone, the communications officer loses contact with the crippled vessel. Klingon starships then appear on an intercept course. Attempts to contact them are met with radio silence; indeed, their only response is to open fire with devastating results. There is no way to win the resulting battle, especially since the computer is allowed to “cheat” to guarantee defeat; the simulation ends with the understanding that the cadet’s ship has been lost with all hands. The objective of the test is not for the cadet to outfight the opponent but rather to test the cadet’s reaction to a “no-win” situation.

Star Fleet believed that only two options were available to the Cadet in command. These were to attempt to save the passengers and risk death along with possibly starting a new war with the Klingon’s, or to abandon the Kobayashi Maru passengers and leave them to die, but in the process save the lives of the Star Fleet crew.

On Cadet James T. Kirk’s third attempt however he proved that this was not the case. Kirk reprogrammed the simulation so that the shields of all 5 of the Klingon ships would drop at the same time, allowing them to be destroyed with ease and thus allowing all civilians on the Kobayashi Maru to be saved.

While you may consider this cheating sometimes an exploratory mind such as Kirk’s in this case can identify a solution which others may not have considered. As testers we need to be capable of identifying ALL possible scenarios for interacting with any system even if the end user is “cheating”. In testing people often say “No user would ever do that!” so why test for that. It is important however as a tester to understand when to “cheat”, and how to “cheat” in order to understand the risks associated with any given system in the given context.

The following is a video which shows Kirk’s third attempt at the Kobayashi Maru test.

Here is what happens when you don’t use the skills of an exploratory tester when taking the exam.

Questions raised for the testing craft from The Kobayashi Maru evaluation:

1. Realistic Simulation:

This is not a multiple choice exam. This is not a written exam. The cadet under evaluation must communicate with real people such as the ships crew and also the enemy ship commander in order to achieve the necessary goals.

2. Persistance:

Kirk took the exam 3 times! It’s important to remember as a tester that sometimes no matter how good you are, you wont get all the information you need on your first try. A true test explorer will try as many times as necessary to get the job done properly. No more no less. Each time learning from the experience.

3. Peer Review:

The result of the evaluation is decided by real Star Fleet captains watching the cadet performing in the role of Capitan. This allows the cadets performance to be evaluated from the perspectives of multiple different experienced Captains who have faced real world challenges. In testing I wonder if a test project simulator were created who would we use to perform the pier reviews and how would we remove the influence of money? But the idea of using respected professionals from the testing community to perform evaluations on testers performing real work on test simulators remains an ideal situation.