Using Multi-Threaded Tests

Author:Matt Albrecht

The GroboUtils class MultiThreadedTestRunner was based on the article "JUnit Best Practices" by Andy Schneider (andrew.schneider@javaworld.com), published online at http://www.javaworld.com/javaworld/jw-12-2000/jw-1221-junit_p.html. Since GroboUtils first started using that implementation, many changes have occured in the code to make a more robust and stable testing environment. Due to these changes, the use of the class will be fully described in this document.

Let's start by testing a sample application:

   1:import java.io.*;
   2:public class WriterCache
   3:{
   4:    public static interface IWriterFactory
   5:    {
   6:        public Writer createWriter( String location )
   7:            throws IOException;
   8:    }
   9:    
  10:    private String openLocation;
  11:    private Writer w;
  12:    private IWriterFactory factory;
  13:    private volatile int openCount = 0;
  14:    
  15:    public WriterCache( IWriterFactory wf )
  16:    {
  17:        if (wf == null)
  18:        {
  19:            throw new IllegalArgumentException( "factory cannot be null" );
  20:        }
  21:        this.factory = wf;
  22:    }
  23:    
  24:    public void writeToFile( String location, String text )
  25:        throws IOException
  26:    {
  27:        if (location == null) return;
  28:        if (!location.equals( this.openLocation ))
  29:        {
  30:            if (this.w != null)
  31:            {
  32:                --this.openCount;
  33:                this.w.close();
  34:            }
  35:            ++this.openCount;
  36:            this.w = this.factory.createWriter( location );
  37:        }
  38:        this.w.write( text );
  39:    }
  40:    
  41:    public int getNumberWritersOpen()
  42:    {
  43:        return this.openCount;
  44:    }
  45:}

Obviously, this class isn't very thread safe - w could easily be closed in one thread when a thread execution switch causes another thread to run its write statement on the closed writer, or even writing to a different writer than what was created (this is data corruption, which is worse than a raised exception).

Note that this class is designed for testing: we can easily create streams in our tests and note what gets written and to which stream. This follows a pattern of designing for mock objects.

Beginning the Unit Test Class

The test class starts off with the standard naming conventions and constructor, and all necessary imports:

 1:import net.sourceforge.groboutils.junit.v1.MultiThreadedTestRunner;
 2:import net.sourceforge.groboutils.junit.v1.TestRunnable;
 3:import junit.framework.TestCase;
 4:import junit.framework.TestSuite;
 5:import java.io.*;
 6:import java.util.*;
 7:
 8:public class WriterCacheUTest extends TestCase
 9:{
10:    public WriterCacheUTest( String name )
11:    {
12:        super( name );
13:    }

Next, we need a way to archive the WriterCache created streams in our own IWriterFactory:

15:    public static class MyWriterFactory implements WriterCache.IWriterFactory
16:    {
17:        Hashtable nameToStream = new Hashtable();
18:        public Writer createWriter( String location )
19:            throws IOException
20:        {
21:            StringWriter sw = (StringWriter)nameToStream.get( location );
22:            if (sw == null)
23:            {
24:                sw = new StringWriter();
25:                nameToStream.put( location, sw );
26:            }
27:            return sw;
28:        }
29:    }
30:}

It uses StringWriter instances so that the tests can examine what what written to which stream.

Ignoring the standard non-threaded tests for such a class (for brevity's sake), we move onto the threaded tests. But before we can write the tests, we need to devise the different tests that we can perform.

One thread can write to one location with a known text pattern, while another writes to a different location with a different text pattern. The test would check to make sure that each location contains only the expected text pattern, and exactly the number of text patterns actually written. Let's call this test "TextPattern"
Use lots of writing threads (say 10), but this time we check for the number of opened writers to ensure it never becomes negative, and generate some kind of warning if it is not 0 or 1. Let's call this test "OpenWriterCount".

Note: since this class uses a StringWriter instance as the returned Writer, the WriterCache calls to close() on the instance will be ignored, as this is the documented behavior of the StringWriter class. Thus, the same StringWriter instance can be sent to the WriterCache over and over.

Creating a TestRunnable

In order to support the above tests, we'll need a way to generate the text pattern for a given location, a given number of times. This is where the GroboUtil extension for multi-threaded testing comes into play. We create another inner class to generate the WriterCache calls by extending the TestRunnable class:

31:    static class WriteText extends TestRunnable
32:    {
33:        private WriterCache wc;
34:        private int count;
35:        private String textPattern;
36:        private String location;
37:        private int sleepTime;
38:        public WriteText( WriterCache wc, int count, String pattern,
39:            String loc, int delay )
40:        {
41:            this.wc = wc;
42:            this.count = count;
43:            this.textPattern = pattern;
44:            this.location = loc;
45:            this.sleepTime = delay;
46:        }
47:        
48:        public void runTest() throws Throwable
49:        {
50:            for (int i = 0; i < this.count; ++i)
51:            {
52:                Thread.sleep( this.sleepTime );
53:                this.wc.writeToFile( this.location, this.textPattern );
54:            }
55:        }
56:    }
57:}

The void runTest() method must be implemented by concrete subclasses of TestRunnable: this is the equivalent of the standard Java interface Runnable's void run() method, but properly wrapped for testing.

Running Several Tasks In Parallel

This allows us to begin writing the test for "TextPattern". This test says that we need to write to two different streams with different text in parallel. Then, we need to ensure that the correct amount of data was written to each, and that no inappropriate data was written. In this test, we'll have one WriteText instance write to location "0" with text "0" (we'll call this thread 0), and another write to location "1" with text "1" (we'll call this thread 1). To vary things up, thread 0 will write 10 times with a 50 millisecond wait, and thread 1 will write 12 times with a 20 millisecond wait. The test ends up looking like:

59:    public void testTextPattern() throws Throwable
60:    {
61:        MyWriterFactor mwf = new MyWriterFactory();
62:        WriterCache wc = new WriterCache( mwf );
63:        TestRunnable tcs[] = {
64:                new WriteText( wc, 10, "0", "0", 50 ),
65:                new WriteText( wc, 12, "1", "1", 20 )
66:            };
67:        MultiThreadedTestRunner mttr =
68:            new MultiThreadedTestRunner( tcs );
69:        mttr.runTestRunnables( 2 * 60 * 1000 );
70:        String s0 = mwf.nameToStream.get( "0" ).toString();
71:        String s1 = mwf.nameToStream.get( "1" ).toString();
72:        assertEquals( "Data corruption: stream 0", 10,
73:            s0.length() );
74:        assertEquals( "Data corruption: stream 1", 12,
75:            s1.length() );
76:        assertTrue( "Incorrect data written to stream 0.",
77:            s0.indexOf( '1' ) < 0 );
78:        assertTrue( "Incorrect data written to stream 1.",
79:            s1.indexOf( '0' ) < 0 );
80:    }
81:

Lines 61-62 initialize the WriterCache instance with our test factory. Line 63 creates our list of parallel tasks to execute, which are instances of our WriteText class above.

Line 64 creates an instance of the MultiThreadedTestRunner class, a utility class which handles the creation, execution, and termination of the threads which run the TestRunnable instances. Line 65 invokes the utility method to run the tasks. The argument to this method specifies the maximum amount of time (in milliseconds) to let the threads run before killing them and marking the execution as a failure (through junit.famework.Assert). If any of the TestRunnable instances dies due to an exception (including an Assert failure), then all of the running threads are terminated, and the runTestRunnables method rethrows the underlying exception.

The remainder of the test ensures that only the correct data was placed where it was intended to go.

Parallel Monitoring of Threaded Access

Now we can move on to the OpenWriterCount test. As of GroboTestingJUnit version 1.2.0, there is a very simple way to add a thread that monitors the status of the common object while other threads manipulate its state.

The monitors are in a separate group from the runners, as the monitors are intended to loop over a set of checks until all the runners have completed. Before this functionality was added, the monitor runners had to communicate with the other runners to learn when the runners were finished. Now, the MultiThreadedTestRunner class handles this communication.

We create another runnable class, but this time subclassing from TestMonitorRunnable. This frees the new class from having to perform the proper looping and detection of when the runners have completed. In order to use this added functionality, the subclasses overload the runMonitor() method instead of the runTest() method.

83:    public static class WriterCountMonitor extends TestMonitorRunnable
84:    {
85:        private WriterCache wc;
86:        public WriterCountMonitor( WriterCache wc )
87:        {
88:            this.wc = wc;
89:        }
90:        
91:        public void runMonitor() throws Throwable
92:        {
93:            int open = this.wc.getNumberWritersOpen();
94:            assertTrue(
95:                "Invalid number of open writers.",
96:                open == 0 || open == 1 );
97:        }
98:    }
99:

The monitors are added to the MultiThreadedTestRunner through a second argument in another constructor. For our test, in order to have high confidence that threading errors causing the writer open count to be invalid (not 1 or 0), we need to have a sufficient number of threaded access on the object-under-test. So, we generate 30 runners to iterate 500 times each over the writer.

101:    public void testOpenWriterCount() throws Throwable
102:    {
103:        int runnerCount = 30;
104:        int iterations = 500;
105:        MyWriterFactor mwf = new MyWriterFactory();
106:        WriterCache wc = new WriterCache( mwf );
107:        TestRunnable tcs[] = new TestRunnable[ runnerCount ];
108:        for (int i = 0; i < runnerCount; ++i)
109:        {
110:            tcs[i] = new WriteText( wc, 500, ""+(char)i,
111:                ""+(char)i, 50 );
112:        }
113:        TestRunnable monitors[] = {
114:                new WriterCountMonitor( wc );
115:            };
116:        MultiThreadedTestRunner mttr =
117:            new MultiThreadedTestRunner( tcs, monitors );
118:        mttr.runTestRunnables( 10 * 60 * 1000 );
119:        
120:        // verify streams
121:        for (int i = 0; i < runnerCount; ++i)
122:        {
123:            String s = mwf.nameToString.get( ""+(char)i ).toString();
124:            assertEquals( "Data corruption: stream "+i,
125:                500, s.length() );
126:        }
127:    }
128:

So, while this test runs the 30 runners, each writing 500 times, the monitor runs concurrently, analyzing the status of the WriterCache instance to ensure its integrety.

As one would expect, both of these tests expose serious synchronization flaws in the WriterCache implementation.

Things To Look Out For

This package isn't without its pitfalls. Here's a checklist to review to ensure that your tests comply with the MTTR caveats.

Never directly run TestRunnable instances. They are designed to only be run in threads generated by the runTestRunnables() method inside the MultiThreadedTestRunner class.
The TestRunnable subclasses need to have their runTest() methods be watchful of InterruptedException and Thread.currentThread().isInterrupted(). The MultiThreadedTestRunner prematurely halts the runner threads by calling Thread.interrupt() on them. If the threads don't terminate themselves within a certain time limit, then MultiThreadedTestRunner will perform the dangerous Thread.stop() operation, in order to prevent threads from going rogue. (Future versions may allow the Thread.stop() call to be disabled.)

Alternatives

One alternative design to this approach is Greg Vaughn's TestDecorator approach, which is located here. However, it forces the decorated Test instance to create the threads itself, and does nothing to manage runaway threads.


This space graciously provided by the SourceForge project	Copyright © 2002-2004 GroboUtils Project. All rights reserved.