August 2010

Volume 25 Number 08

Test Run - Fault Injection Testing with TestApi

By James McCaffrey | August 2010

Fault injection testing is the process of deliberately inserting an error into an application under test and then running the application to determine whether the application deals with the error properly. Fault injection testing can take several different forms. In this month’s column, I explain how you can introduce faults into .NET applications at run time using a component of the TestApi library.

The best way for you to see where I’m headed in this column is to take a look at the screenshot in Figure 1. The screenshot shows that I’m performing fault injection testing on a dummy .NET WinForm application named TwoCardPokerGame.exe. A C# program named FaultHarness.exe is running in the command shell. It alters the normal behavior of the application under test so the application throws an exception the third time a user clicks on the button labeled Evaluate. In this situation, the Two Card Poker application does not handle the application exception gracefully and the result is the system-generated message box.

Figure 1  Fault Injection Testing in Action

Let’s take a closer look at this scenario to consider some of the details involved. When FaultHarness.exe is launched from the command shell, behind the scenes the harness prepares profiling code that will intercept the normal code execution of TwoCard­PokerGame.exe. This is called the fault injection session.

The fault injection session uses a DLL to start watching for calls to the application’s button2_Click method, which is the event handler for the button labeled Evaluate. The fault injection session has been configured so that the first two times a user clicks on the Evaluate button, the application behaves as coded, but on the third click the fault session causes the application to throw an exception of type System.ApplicationException.

The fault session records session activity and logs a set of files to the test host machine. Notice in Figure 1 that the first two application Deal-Evaluate click pairs work properly, but the third click generated an exception.

In the sections that follow, I’ll briefly describe the dummy Two Card Poker Game application under test, present and explain in detail the code in the FaultHarness.exe program shown in Figure 1, and provide some tips about when the use of fault injection testing is appropriate and when alternative techniques are more suitable. Although the FaultHarness.exe program itself is quite simple and most of the difficult work is performed behind the scenes by the TestApi DLLs, understanding and modifying the code I present here to meet your own testing scenarios requires a solid understanding of the .NET programming environment. That said, even if you’re a .NET beginner, you should be able to follow my explanations without too much difficulty. I’m confident you’ll find the discussion of fault injection an interesting and possibly useful addition to your toolset.

The Application Under Test

My dummy application under test is a simplistic but representative C# WinForm application that simulates a hypothetical card game called Two Card Poker. The application consists of two main components: TwoCardPokerGame.exe provides the UI and TwoCardPokerLib.dll provides the underlying functionality.

To create the game DLL I launched Visual Studio 2008 and selected the C# Class Library template from the File | New Project dialog box. I named the library TwoCardPokerLib. The overall structure of the library is presented in Figure 2. The code for TwoCardPokerLib is too long to present in its entirety in this article. The complete source code for the TwoCardPokerLib library and the FaultHarness fault injection harness is available in the code download that accompanies this article.

Figure 2 The TwoCardPokerLib Library

using System;
namespace TwoCardPokerLib {
  // -------------------------------------------------
  public class Card {
    private string rank;
    private string suit;
    public Card() {
      this.rank = "A"; // A, 2, 3, . . ,9, T, J, Q, K
      this.suit = "c"; // c, d, h, s
    }
    public Card(string c) { . . . }
    public Card(int c) { . . . }
    public override string ToString(){ . . . }
    public string Rank { . . . }
    public string Suit { . . . }
    public static bool Beats(Card c1, Card c2) { . . . }
    public static bool Ties(Card c1, Card c2) { . . . }
  } // class Card
  // -------------------------------------------------
  public class Deck {
    private Card[] cards;
    private int top;
    private Random random = null;
    public Deck() {
      this.cards = new Card[52];
      for (int i = 0; i < 52; ++i)
        this.cards[i] = new Card(i);
      this.top = 0;
      random = new Random(0);
    }
    public void Shuffle(){ . . . }
    public int Count(){ . . . } 
    public override string ToString(){ . . . }
    public Card[] Deal(int n) { . . . }
    
  } // Deck
  // -------------------------------------------------
  public class Hand {
    private Card card1; // high card
    private Card card2; // low card
    public Hand(){ . . . }
    public Hand(Card c1, Card c2) { . . . }
    public Hand(string s1, string s2) { . . . }
    public override string ToString(){ . . . }
    private bool IsPair() { . . . }
    private bool IsFlush() { . . . }
    private bool IsStraight() { . . . }
    private bool IsStraightFlush(){ . . . }
    private bool Beats(Hand h) { . . . }
    private bool Ties(Hand h) { . . . }
    public int Compare(Hand h) { . . . }
    public enum HandType { . . . }
    
 } // class Hand
} // ns TwoCardPokerLib

The Application UI Code

Once I had the underlying TwoCardPokerLib library code finished, I created a dummy UI component. I started a new project in Visual Studio 2008 using the C# WinForm Application template and I named my application TwoCardPokerGame.

Using the Visual Studio designer, I dragged a Label control from the Toolbox collection onto the application design surface, and modified the control’s Text property from “textBox1” to “Two Card Poker.” Next I added two more Label controls (“Your Hand” and “Computer’s Hand”), two TextBox controls, two Button controls (“Deal” and “Evaluate”), and a ListBox control. I didn’t change the default control names of any of the eight controls—textBox1, textBox2, button1 and so on.

Once my design was in place, I double-clicked on the button1 control to have Visual Studio generate an event handler skeleton for the button and load file Form1.cs into the code editor. At this point I right-clicked on the TwoCardPokerGame project in the Solution Explorer window, selected the Add Reference option from the context menu, and pointed to the file TwoCardPokerLib.dll. In Form1.cs, I added a using statement so that I wouldn’t need to fully qualify the class names in the library.

Next, I added four class-scope static objects to my application:

namespace TwoCardPokerGame {
  public partial class Form1 : Form {
    static Deck deck;
    static Hand h1;
    static Hand h2;
    static int dealNumber; 
...

Object h1 is the Hand for the user, and h2 is the Hand for the computer. Then I added some initialization code to the Form constructor:

public Form1() {
  InitializeComponent();
  deck = new Deck();
  deck.Shuffle();
  dealNumber = 0;
}

The Deck constructor creates a deck of 52 cards, in order from the ace of clubs to the king of spades, and the Shuffle method
randomizes the order of the cards in the deck.

Next I added the code logic to the button1_Click method as shown in Figure 3. For each of the two hands, I call the Deck.Deal method to remove two cards from the deck object. Then I pass those two cards to the Hand constructor and display the value of the hand in a TextBox control. Notice that the button1_Click method handles any exception by displaying a message in the ListBox control.

Figure 3 Dealing the Cards

private void button1_Click(
  object sender, EventArgs e) { 
  try  {
    ++dealNumber;
    listBox1.Items.Add("Deal # " + dealNumber);
    Card[] firstPairOfCards = deck.Deal(2);
    h1 = new Hand(firstPairOfCards[0], firstPairOfCards[1]);
    textBox1.Text = h1.ToString();
    Card[] secondPairOfCards = deck.Deal(2);
    h2 = new Hand(secondPairOfCards[0], secondPairOfCards[1]);
    textBox2.Text = h2.ToString();
    listBox1.Items.Add(textBox1.Text + " : " + textBox2.Text);
  }
  catch (Exception ex) {
    listBox1.Items.Add(ex.Message);
  }
}

Next, in the Visual Studio designer window I double-clicked on the button2 control to auto-generate the control’s event handler
skeleton. I added some simple code to compare the two Hand objects and display a message in the ListBox control. Notice that the button2_Click method does not directly handle any exceptions:

private void button2_Click(
  object sender, EventArgs e) {
  int compResult = h1.Compare(h2);
  if (compResult == -1)
    listBox1.Items.Add(" You lose");
  else if (compResult == +1)
    listBox1.Items.Add(" You win");
  else if (compResult == 0)
    listBox1.Items.Add(" You tie");
  listBox1.Items.Add("-------------------------");
}

The Fault Injection Harness

Before creating the fault injection harness shown in Figure 1, I downloaded the key DLLs to my test host machine. These DLLs are part of a collection of .NET libraries named TestApi and can be found at testapi.codeplex.com.

The TestApi library is a collection of software-testing-related utilities. Included in the TestApi library is a set of Managed Code Fault Injection APIs. (Read more about them at blogs.msdn.com/b/ivo_manolov/archive/2009/11/25/9928447.aspx.) I downloaded the latest fault injection APIs release, which in my case was version 0.4, and unzipped the download. I will explain what’s in the download and where to place the fault injection binaries shortly.

Version 0.4 supports fault injection testing for applications created using the .NET Framework 3.5. The TestApi library is under active development, so you should check the CodePlex site for updates to the techniques I present in this article. Additionally, you may want to check for updates and tips on the blog of Bill Liu, the primary developer of the TestApi fault injection library, at blogs.msdn.com/b/billliu/.

To create the fault injection harness I started a new project in Visual Studio 2008 and selected the C# Console Application template. I named the application FaultHarness and I added some minimal code to the program template (see Figure 4).

Figure 4 FaultHarness

using System;
namespace FaultHarness {
  class Program {
    static void Main(string[] args) {
      try {
        Console.WriteLine("\nBegin TestApi Fault Injection environmnent session\n");
        // create fault session, launch application
        Console.WriteLine("\nEnd TestApi Fault Injection environment session");
      }
      catch (Exception ex) {
        Console.WriteLine("Fatal: " + ex.Message);
      }
    }
  } // class Program
} // ns

I hit the <F5> key to build and run the harness skeleton, which created a \bin\Debug folder in the FaultHarness root folder.

The TestApi download has two key components. The first is TestApiCore.dll, which was located in the Binaries folder of the unzipped download. I copied this DLL into the root directory of the FaultHarness application. Then I right-clicked on the FaultHarness project in the Solution Explorer window, selected Add Reference, and pointed it to TestApiCore.dll. Next, I added a using statement for Microsoft.Test.FaultInjection to the top of my fault harness code so my harness code could directly access the functionality in TestApiCore.dll. I also added a using statement for System.Diagnostics because, as you’ll see shortly, I want to access the Process and ProcessStartInfo classes from that namespace.

The second key component in the fault injection download is a folder named FaultInjectionEngine. This holds 32-bit and 64-bit versions of FaultInjectionEngine.dll. I copied the entire Fault­InjectionEngine folder into the folder holding my FaultHarness executable, in my case C:\FaultInjection\FaultHarness\bin\Debug\. The 0.4 version of the fault injection system I was using requires the FaultInjectionEngine folder to be in the same location as the harness executable. Additionally, the system requires that the application under test binaries be located in the same folder as the harness executable, so I copied files TwoCardPokerGame.exe and TwoCard­PokerLib.dll into C:\FaultInjection\FaultHarness\bin\Debug\.

To summarize, when using the TestApi fault injection system, a good approach is to generate a skeleton harness and run it so that a harness \bin\Debug directory is created, then place file TestApiCore.dll in the harness root directory, place the FaultInjectionEngine folder in \bin\Debug, and place the application under test binaries (.exe and .dll) in \bin\Debug as well.

Using the TestApi fault injection system requires that you specify the application under test, the method in the application under test that will trigger a fault, the condition that will trigger a fault, and the kind of fault that will be triggered:

string appUnderTest = "TwoCardPokerGame.exe";
string method = 
  "TwoCardPokerGame.Form1.button2_Click(object, System.EventArgs)";
ICondition condition =
  BuiltInConditions.TriggerEveryOnNthCall(3);
IFault fault =
  BuiltInFaults.ThrowExceptionFault(
    new ApplicationException(
    "Application exception thrown by Fault Harness!"));
FaultRule rule = new FaultRule(method, condition, fault);

Notice that, because the system requires the application under test to be in the same folder as the harness executable, the name of the application under test executable does not need the path to its location.

Specifying the name of the method that will trigger the injected fault is a common source of trouble for TestApi fault injection beginners. The method name must be fully qualified in the form Name­space.Class.Method(args). My preferred technique is to use the ildasm.exe tool to examine the application under test to help me determine the triggering method’s signature. From the special Visual Studio tools command shell I launch ildasm.exe, point to the application under test, then double-click on the target method. Figure 5 shows an example of using ildasm.exe to examine the signature for the button2_Click method.

Figure 5 Using ILDASM to Examine Method Signatures

When specifying the trigger method signature, you do not use the method return type, and you do not use parameter names. Getting the method signature correct sometimes requires a bit of trial and error. For example, on my first attempt to target button2_Click, I used:

TwoCardPokerGame.Form1.button2_Click(object,EventArgs)

I had to correct it to:

TwoCardPokerGame.Form1.button2_Click(object,System.EventArgs)

The TestApi download contains a Documentation folder containing a concepts document that provides good guidance on how to correctly construct different kinds of method signatures including constructors, generic methods, properties, and overloaded operators. Here I target a method that’s located in the application under test, but I could have also targeted a method in the underlying Two­CardPokerLib.dll, such as:

string method = "TwoCardPokerLib.Deck.Deal(int)"

After specifying the trigger method, the next step is to specify the condition under which the fault will be injected into the application under test. In my example I used TriggerEveryOnNthCall(3), which as you’ve seen injects a fault every third time the trigger method is called. The TestApi fault injection system has a neat set of trigger conditions including TriggerIfCalledBy(method), TriggerOnEveryCall, and others.

After specifying the trigger condition, the next step is to specify the type of fault that will be injected into the system under test. I used BuiltInFaults.ThrowExceptionFault. In addition to exception faults, the TestApi fault injection system has built-in return type faults that allow you to inject erroneous return values into your application under test at run time. For example, this will cause the trigger method to return a (presumably incorrect) value of -1:

IFault f = BuiltInFaults.ReturnValueFault(-1)

After the fault trigger method, condition, and fault kind have been specified, the next step is to create a new FaultRule and pass that rule to a new FaultSession:

FaultRule rule = new FaultRule(method, condition, fault);
Console.WriteLine(
  "Application under test = " + appUnderTest);
Console.WriteLine(
  "Method to trigger injected runtime fault = " + method);
Console.WriteLine(
  "Condition which will trigger fault = On 3rd call");
Console.WriteLine(
  "Fault which will be triggered = ApplicationException");
FaultSession session = new FaultSession(rule);

With all the preliminaries in place, the last part of writing the fault harness code is to programmatically launch the application under test in the fault session environment:

ProcessStartInfo psi = 
  session.GetProcessStartInfo(appUnderTest);
Console.WriteLine(
  "\nProgrammatically launching application under test");
Process p = Process.Start(psi);
p.WaitForExit();
p.Close();

When you execute the fault harness, it will launch the application under test in your fault session, with the FaultInjection­Engine.dll watching for situations where the trigger method is called when the trigger condition is true. The tests are performed manually here, but you can also run test automation in a fault session.

While the fault session is running, information about the session is logged into the current directory—that is, the directory that holds the fault harness executable and the application under test executable. You can examine these log files to help resolve any problems that might occur while you’re developing your fault injection harness.

Discussion

The example and explanations I've presented here should get you up and running with creating a fault injection harness for your own application under test. As with any activity that’s part of the software development process, you will have limited resources and you should analyze the costs and benefits of performing fault injection testing. In the case of some applications, the effort required to create fault injection testing may not be worthwhile, but there are many testing scenarios where fault injection testing is critically important. Imagine software that controls a medical device or a flight system. In situations such as these, applications absolutely must be robust and able to correctly handle all kinds of unexpected faults.

There is a certain irony involved with fault injection testing. The idea is that, if you can anticipate the situations when an exception can occur, you can in theory often programmatically guard against that exception and test for the correct behavior of that guarding behavior. However, even in such situations, fault injection testing is useful for generating difficult to create exceptions. Additionally, it’s possible to inject faults that are very difficult to anticipate, such as System.OutOfMemoryException.

Fault injection testing is related to and sometimes confused with mutation testing. In mutation testing, you deliberately insert errors into the system under test, but then execute an existing test suite against the faulty system in order to determine whether the test suite catches the new errors created. Mutation testing is a way to measure test suite effectiveness and ultimately increase test case coverage. As you’ve seen in this article, the primary purpose of fault injection testing is to determine whether the system under test correctly handles errors.

Dr. James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at the Microsoft Redmond, Wash., campus. He’s worked on several Microsoft products, including Internet Explorer and MSN Search. Dr. McCaffrey is the author of “.NET Test Automation Recipes” (Apress, 2006) and can be reached at [email protected].

Thanks to the following technical experts for reviewing this article: Bill Liu and Paul Newson