Tools

Digg This!

Page 2 of 2   « previous page

Finite State Machines (FSM)
Once you have a model put together and you've built it, you have to tell Spec Explorer that its "Exploration Goal is FSM generation." You can find that command on the Explore menu. Open the Channel.doc example that comes with Spec Explorer and try it out. First, choose Build from the Project menu, and then choose "Exploration Goal is FSM Generation" from the Explore menu.

A finite state machine is really one instance of your model space. Spec Explorer can generate new and different finite state machines. You just have to give it a new random seed under Exploration Settings on the Explore menu. The FSM will be different with a new random seed if it were limited somehow - for example, if you limited the finite state machine by using state grouping or limits on the number of transitions or states to explore. If the FSM is complete, that is, without limits, then it will represent all of the states and transitions so that the RandomSeed value won't matter.

Spec Explorer automatically generates a view of your model when you choose Run from the Explore Menu (see Figure 2). The Channel example has two views: Full and Grouped. You can create as many views as you want of your model. You'll find that the grouping feature is not only handy but becomes necessary as your state space starts to grow bigger. You would group by some model expression that is important in the context of your model. The Channel sample is grouped by OrderExpression and the expression looks like this:

Bag{c in enumof(Channel); c.state}

Figure 3 provides the same grouped view by itself for ease of reading. Spec Explorer allows you to save off the graphical views of your model. I often include these in my own test specs when I'm developing models and save them as JPGs. They are handy for reference.

Generating Test Suites
Once you have an explorable model, you're ready to generate test suites. But wait! You have to implement something first, preferably a class library. The actions you specify in your model spec must have the same function signature and return type as the methods of the test class(es) that you implement in your class library. In our previous example, we had an action that resembled:

void LogOn(string userName, string password)

That means in your class library you should have a publicly accessible method of a class with the same function signature. If you choose Test Settings from the Test menu, you bring up a page and choose the Test Bindings tab (see Figure 4).

If you click the Autofill Scope button, you can drill down into your class library's namespace and choose the class name. Spec Explorer will automatically bind your class methods to your model actions.

Once the test bindings are in place, you may generate test suites to your heart's content. The Run command on the Test menu will actually run your automation. If you need to export your test automation so that it may be run from the command line, I have written a few simple procedures below.

To Generate a Test Suite

1. From the Project Menu of Spec Explorer, choose Build to build your model.
2. If your model builds without errors, you're free to choose the Run command from the Explore menu. The Exploration Goal should be set to FSM (for finite state machine). Spec Explorer will generate an instance of a finite state machine for your model space.
3. From the Test menu, select the Generate Test Suites command. Note that to do this, you must have already added a test suite to the project. That's fairly straight-forward (right-click on the Test Suites folder and choose Add).
4. From the Test menu, choose Generate Code for Test Suites. That will generate a Visual Studioproject with accompanying CSharp file that will hold your test automation.

To Run Test Cases in an XML File from the Command Line

1. Make sure the generated code from the last procedure is built. Opening the project in Visual Studio and choosing Build will create the necessary executable.
2. Run the following command. The name "TestSuites.exe" is the name of the test harness:

   
   TestSuites.exe /TestSuites:<TestSuiteNameHere>
   

   or alternatively, if you want to run a specific case:

   
   TestSuites.exe /segments:<name>=<segment numbers>
   

   where the parameter <name> is the name of a test segment you wish to run, and "TestSuites.exe" is the name of the test executable built. The key here is the segment numbers, which you can get from Spec Explorer when you generate test suites in the IDE.

Spec Explorer's On-the-Fly Testing
I don't want to finish this article without saying something about on-the-fly testing. The real power behind model-based testing lies in allowing the model automation to run and explore the space. If you choose Start On the Fly Testing from the Tools menu, Spec Explorer will run your model automation and hopefully you'll be filing new bugs in no time. It's not about creating test cases for their own sake, it's about finding bugs - so let the model automation find the bugs for you. On-the-fly testing is powerful when: 1) the model space is huge, 2) many transitions are unlikely, or 3) the system is asynchronous. On-the-fly testing is not guaranteed to explore all possible paths, but it is good at quickly exploring common ones.

Since number 1 is obvious, let's consider the second case: many transitions are unlikely. Suppose you have a function bool Foo() that writes to a file and returns success or failure. It may return false if the hard drive is out of space. Full FSM generation will explore both cases, but in reality you may never see a failure because most hard drives are large. On-the-fly testing won't waste time analyzing the path where Foo failed if the hard drive never runs out of space at run time.

Now consider possibility 3: the system is asynchronous. Suppose your implementation has multiple threads, and when you call Foo() it causes two observable events, Observable1() and Observable2(), to happen asynchronously. Full FSM generation will explore both orderings (Observable1 then Observable2, and vice versa), but in reality the NT thread scheduler may cause (1 then 2) 99 percent of the time. On-the-fly testing won't waste the time analyzing the (2 then 1) path 99 percent of the time, because it doesn't happen at run time.

Conclusion
A model-based approach to testing can really make a difference to divisions that are short on personnel. You still have to invest the time in the implementation of test libraries, of course, and you need to invest time in developing good models to represent the system under test. In addition, you should ask the staff responsible for product design to review your models. Thus, model-based testing is a powerful approach to finding bugs. If you're worried about the traditional organizational metrics, focus on code coverage and bug count. Remember, it's not the number of test cases you have developed, it's the breadth and depth of your testing and the bugs that are found that matter.

Page 2 of 2   « previous page