Back when I was a PhD student at McGill, our lab had a rule of thumb when it came to robot tests: if the robot can't repeat the test at least ten times then it can't do it. You can see this in Dave McMordie's thesis, in Ned Moore's thesis, in Neil Neville's thesis and in mine -- at least ten trials, but often 20, 40, 90 or more. We're looking for meaningful results that are robust to changes in test conditions.
Since the introduction of jUnit 5 we have been able to easily perform repeated tests. And, for me, knowing that I am likely to run tests on hardware systems (like robots and sensors) or non-deterministic processes like LLMs, I think that it's important to be able to tie these repeated tests to some notion of mean and standard deviation
With that in mind, I've been struggling to figure out a way to save state with jUnit with the goal of running multiple tests and averaging out the outputs. I thought that maybe I could use logging, but that's really one-way street. Extensions held out promise, but initial attempts didn't work and seem to make things more complicated than they should. Stack overflow added an extra hint: reflection. I've never used reflection before. These all sound more complicated than they should.
And, really, I was complicating my life. All I needed was a class-wide variable and to realize that jUnit can invoke, by default, each test method in its own class instance.
Again, one of the tricky things to realize with jUnit is that it runs separately from your regular java classes. This brings up the notion of "life-cycle" "life cycle" of testing. By default, jUnit's life-cycle is on a "per method" basis -- effectively think that each method in a test class runs independently and does not rely on any other test method in the same test class. So if one test method produces something no other test method will know about it.
So, in order to keep persistent track of test values (state) I need to keep in mind how jUnit runs its test methods and treats its classes.
Test Methods Run in New Class Instances
By default, each method in a test class will execute independently of one-another. It appears that each time a test method is set to run, a new instance of the class will be created before the method executes. Counter-intuitively, this is as true for the @RepeatedTest as it is for @Test annotations.
Retaining State within a Class using Static Variable
Ex: Non-static counting variable
This can be demonstrated by setting up a test class with a class-scoped counter called "runningSum":
// junit6TestVsRepeated Project
// TestCompareRepeated.java
import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.Test;
public class TestCompareRepeated {
Integer runningSum = 0;
@Test
void testA(){
runningSum = runningSum + 1;
System.out.println("TestA: running sum: " + runningSum);
}
@Test
void testB(){
runningSum = runningSum + 1;
System.out.println("TestB: running sum: " + runningSum);
}
@Test
void testC(){
runningSum = runningSum + 1;
System.out.println("TestC: running sum: " + runningSum);
}
@RepeatedTest(5)
void testD(){
runningSum = runningSum + 1;
System.out.println("TestD: running sum: " + runningSum);
}
}
The result will look like this:
Ex: Static counting variable
Now, if I convert the running sum variable to a static value, meaning that its scope applies to all instances of the test class:
static Integer runningSum = 0; Then the result is this:
Retaining State within a Class using LifeCycle Annotation
Ex: Per-Class LifeCycle
You can achieve the same effect as setting the counter to be static if you make jUnit change from it's default "per method" lifecycle to a "per class lifecycle". This is achieved by putting a @TestInstance annotation before the class signature and specifying within the input argument of the annotation, the kind of lifecycle it is.
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class TestCompareRepeated {I've implemented this here:
// junit6TestVsRepeatedv2 Project
// TestCompareRepeated.java
import org.junit.jupiter.api.*;
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class TestCompareRepeated {
Integer runningSum = 0; // Alternate b/w static and non-static
@Test
void testA(){
runningSum = runningSum + 1;
System.out.println("TestA: running sum: " + runningSum);
}
@Test
void testB(){
runningSum = runningSum + 1;
System.out.println("TestB: running sum: " + runningSum);
}
@Test
void testC(){
runningSum = runningSum + 1;
System.out.println("TestC: running sum: " + runningSum);
}
@RepeatedTest(5)
void testD(){
runningSum = runningSum + 1;
System.out.println("TestD: running sum: " + runningSum);
}
}
I saw this described in the Test Automation University jUnit 5 tutorials here (GitHub).
Problem 1: What if jUnit executes test methods in a different order?
We can force this kind of process that artificially setting the execution order of test methods. Here is a sample program that forces test execution to be in the order written:
// junit6testordereample project
// Source: https://docs.junit.org/6.1.0/writing-tests/test-execution-order.html
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
@TestMethodOrder(OrderAnnotation.class)
class testOrderExample {
@Test
@Order(1)
void testA() {
System.out.println("Test A");
}
@Test
@Order(2)
void testB() {
System.out.println("Test B");
}
@Test
@Order(3)
void testC() {
System.out.println("Test C");
}
}Which results in this output:
Next, let's make "Test C" go first:
// junit6testordereample project
// Source: https://docs.junit.org/6.1.0/writing-tests/test-execution-order.html
import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
@TestMethodOrder(OrderAnnotation.class)
class testOrderExample {
@Test
@Order(3) // Now it's the third test.
void testA() {
System.out.println("Test A");
}
@Test
@Order(2)
void testB() {
System.out.println("Test B");
}
@Test
@Order(1) // Now it's the first test
void testC() {
System.out.println("Test C");
}
}
Which results in the order changing:
Problem 2: What if we need to distinguish between successful and unsuccessful test?
Unreliable Tests
In real life, not every test will succeed. Some will fail. For instance, a robot could crash in the middle of a test run, or a sensor could start transmitting an impossible result, like -80 C for a thermometer that is only rated from 0 to 40. The @RepeatedTest annotation permits you to deal with those by, for instance, invoking a fail() after doing some kind of conditional test. You could also invoke the fail method if an exception occurs.
The following is an example that pretends that a randomly generated value is a stand-in for a sensor value. We want to find the average and standard deviation of all the sensor results.
// junit6TestVsRepeatedv3 Project
// TestCompareRepeated.java
// References:
// 1. Test AUtomation University:
// 2. Fail: https://www.baeldung.com/junit-fail
// 3. Repeated Test extras: https://docs.junit.org/6.1.0/writing-tests/repeated-tests.html
import org.junit.jupiter.api.*;
import java.util.ArrayList;
import java.util.Random;
import static java.lang.Math.sqrt;
import static org.junit.jupiter.api.Assertions.fail;
@TestInstance(TestInstance.Lifecycle.PER_CLASS) // avoids need for static.
public class TestCompareRepeated {
Integer runningSum = 0; // behaves as static because "per class" TestInstance
ArrayList<Double> measuredValues = new ArrayList<>();
Double average = 0.0d;
Double stdev = 0.0d;
Random r = new Random();
// Five repeated tests. Allow for one failed test without stopping.
@RepeatedTest(value = 5, failureThreshold = 2, name = RepeatedTest.LONG_DISPLAY_NAME)
@DisplayName("TestD")
void testD(TestInfo testInfo, RepetitionInfo repetitionInfo){
// Artificially set up a condition for the 3rd test to fail.
if (repetitionInfo.getCurrentRepetition() == 3) {
fail("Boom!");
}else {
runningSum = runningSum + 1;
}
System.out.print(testInfo.getDisplayName()); // Display "display" info.
System.out.println(" running sum: " + runningSum);
/* Calculate average and standard deviation */
measuredValues.add(r.nextDouble());
//average = measuredValues.stream().mapToDouble(s->s).average();
for(Double i : measuredValues){
average = average + i;
}
average = average/measuredValues.toArray().length;
for(Double i : measuredValues){
stdev = (i-average)*(i-average);
}
stdev = sqrt(stdev/measuredValues.toArray().length);
}
// Once all tests are complete, print out the average and standard deviation.
@AfterAll
void wrapUp(){
System.out.println("Average: " + average + " and StdDev: " + stdev);
}
}
Here is the unit test output, showing Test 3 failing.
And we can see that the output of the unit testing in IntelliJ results in four of five successful tests, with the one in the middle (Test 3) fails. But it doesn't stop the testing from continuing. The average and standard deviation calculations continue, even up to the last test.
What I haven't done here: Parallelism & Timeouts
I'm assuming that all of this is running as a single thread. I'm not sure what the effect of parallel execution will be on this. Something to keep in mind for future work. Also, jUnit has a time-out annotation that I should probably work into this example, too.
Conclusion
I've likely missed something here that is obvious to a Java expert but for now, I've got the skeleton of a testing setup that can figure out average and standard deviation over successful runs while also taking failed tests into consideration.
James Andrew Smith is a Professional Engineer and Associate Professor in the Electrical Engineering and Computer Science Department of York University’s Lassonde School, with degrees in Electrical and Mechanical Engineering from the University of Alberta and McGill University. Previously a program director in biomedical engineering, his research background spans robotics, locomotion, human birth, music and engineering education. While on sabbatical in 2018-19 with his wife and kids he lived in Strasbourg, France and he taught at the INSA Strasbourg and Hochschule Karlsruhe and wrote about his personal and professional perspectives. James is a proponent of using social media to advocate for justice, equity, diversity and inclusion as well as evidence-based applications of research in the public sphere. You can find him on Twitter. You can find him on BlueSky. Originally from Québec City, he now lives in Toronto, Canada.