Code Coverage Design (V1)

Author:Matt Albrecht

Overview

The goal of the codecoverage package is to capture the lines of code that were executed in a Java VM. Thanks to the JPDA (Java Platform Debugger Architecture), Java code can now monitor Java VMs to discover which lines of code were executed.

This package breaks the general code-coverage problem into three parts:

tracker: tracks the remote VM as the code is executed. This is written to a file or other data source for later parsing.
counter: discovers the actual number of lines in each class. This output can be written to a file or other data source for later parsing.
collator: parses the tracker and counter output, and creates output which displays the actual number of lines, and those which were covered. This depends upon the other two parts.

This package must be classloader aware. That is, multiple classes with the same name may be loaded in a single VM. Currently, this is handled by calculating the CRC of each class. This not only detects different class files, but also allows the package to detect if there are differences between a tracked class and a counted class.

Definitions

As the code coverage package deals with multiple VMs, I need to establish some lingo such that others, as well as myself, may understand more clearly the ideas I am attempting to state. First, I will state the common design. Then, I will assign definitions to the entities in the design.

The general flow of the application is:

Either the code coverage tool or another tool starts a Java Virtual Machine (VM), which is listening for Java Debug Wire Protocol (JDWP) connections; this VM will execute the Java bytecode which will be analyzed for the code it executes. Let's call this the Remote VM, since it is remote to the code coverage tool.
The code coverage tool is executed within another Virtual Machine. Since this document analyzes the process flow from the perspective of the code coverage tool, let us call this the Local VM.

Current Design Assumptions

The current design assumes that the bytecode retrieved by BCEL is identical to the bytecode that JPDA returns. If this assumption fails, then the class checksums won't match, causing the actual class lines not to be found in the counter.

Recent experiments found that this assumption is wrong. There is a temporary work-around (via ignoring checksums during collation), however a better solution will need to be found.

Current Poorly Designed Parts

The IMethodLinesRecord and IClassLinesRecord both merge the marshalling and unmarshalling code together. It is useful in that one can observe from within the same file that one's logic is the reverse of the other. However, this is a poor design choice in that a code section can't tell if a record can read or write. This needs to be split apart (however, the marshall/unmarshall code can remain in the same class, since it will be implementing interfaces).

Dealing with JPDA Problems

I have found JPDA to be incredibly flakey as of JDK 1.4:

There is a chance, on entry to any native code, that inspection of the StepEvent encountered by the Local VM will cause the Remote VM JPDA implementation will not understand the stack, causing a VM Death in the Remote VM.

I have discovered this to occur commonly in calls to System.exit( int ), but it has also occured within java.util.jar.Attributes$Name.isValid:

     [java] FATAL ERROR in native method: JDWP "stepControl.c" (Feb  7 2002), li
ne 152: Unable to get frame location, error code = 32 (JVMDI_ERROR_OPAQUE_FRAME)

     [java]
     [java]     at java.util.jar.Attributes$Name.isValid(Attributes.java:410)
     [java]     at java.util.jar.Attributes$Name.isValid(Attributes.java:402)
     [java]     at java.util.jar.Attributes$Name.<init>(Attributes.java:390)
     [java]     at java.util.jar.Attributes.putValue(Attributes.java:144)
     [java]     at java.util.jar.Attributes.read(Attributes.java:362)
     [java]     at java.util.jar.Manifest.read(Manifest.java:162)
     [java]     at java.util.jar.Manifest.<init>(Manifest.java:52)
     [java]     at java.util.jar.JarFile.getManifest(JarFile.java:147)
     [java]     at sun.misc.URLClassPath$JarLoader.getClassPath(URLClassPath.jav
a:712)
     [java]     at sun.misc.URLClassPath.getLoader(URLClassPath.java:224)
     [java]     - locked <02A937C8> (a sun.misc.URLClassPath)
     [java]     at sun.misc.URLClassPath.getResource(URLClassPath.java:134)
     [java]     at sun.misc.URLClassPath.getResource(URLClassPath.java:144)
     [java]     at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:8
80)
     [java]     at java.lang.ClassLoader.getResource(ClassLoader.java:784)
     [java]     at java.lang.ClassLoader.getResource(ClassLoader.java:782)
     [java]     at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:94
8)
     [java]     at java.lang.Class.getResourceAsStream(Class.java:1284)
     [java]     at org.apache.tools.ant.Main.getAntVersion(Main.java:564)
     [java]     - locked <06B0E6F8> (a java.lang.Class)
     [java]     at org.apache.tools.ant.Main.printVersion(Main.java:555)
     [java]     at org.apache.tools.ant.Main.<init>(Main.java:200)
     [java]     at org.apache.tools.ant.Main.start(Main.java:138)
     [java]     at org.apache.tools.ant.Main.main(Main.java:176)
     [java] Java Result: 1
     [echo] Stopping JUnit.

There are times when the JPDA events received by the Local VM from the Remote VM do not properly portray the suspended state of the Remote VM (this may actually be a result of insufficiently detailed documentation describing what events actually suspend), causing the Local VM to not receive events.
At times, the StepRequest.setEnabled( true ) method will hang. This is currently under investigation.
The "includes" and "excludes" methods for StepRequests fail to work as described.

Here are the current work-arounds for each part:

The exception is trapped, and interpreted to mean the equivalent of a VM Death notice. Not implemented yet.
If no events have been received over a period of time, then attempts will be made to resume each thread in the Remote VM. "Over a period of time" means that the pull events method does not return any events after a specified number of attempts, and each attempt with a certain time-out period.
A proposed solution will execute the setEnabled call in a separate thread, and kill that thread if it does not return after a period of several seconds. Not implemented yet.

In the future, these should be abstracted out to per-JPDA implementation classes for a better design.

JPDA Bugs of Interest

4490152 - JPDA: vm-wide suspend does not suspend all threads (zombie, dead, new, ...)
In all likelyhood the thread in question is a zombie thread.

HotSpot will return a thread status of unknown and a zero suspend status for threads it doesn't know about (which includes threads that haven't run yet, threads being initialized or threads that have completed running). HotSpot will return a thread status of zombie and a zero suspend status for threads being shutdown. The code of ThreadReference has code in it to do it's own accounting for suspends of zombies (this code is currently broken).

It may be that the correct approach is to spec in all interfaces (JVMDI/JDWP/JDI) that new/zombie threads cannot be suspended (or "may" be incapable of being suspended).


This space graciously provided by the SourceForge project	Copyright © 2002-2004 GroboUtils Project. All rights reserved.