Skip to content

Some Thoughts on Integrating SoapUI Functional Tests with Your Build

June 23, 2010

The development team I’m a part of at Ohio Mutual Insurance Group is about to develop our own web services for the first time and I want to make sure we test them properly. So, for about two months now I have been experimenting with using SoapUI to aid in developing our services.  One very important aspect of my experiment is to determine how difficult it is to run SoapUI functional tests from a Hudson CI job. In this blog are my rough thoughts on how well SoapUI functional tests integrate with the build process we use on Hudson.

Functional testing is not all SoapUI does, but it is one outstanding feature that I’m focusing on for now. Even though SoapUI has a solid interface for running functional tests, it doesn’t make any sense to have compilation, unit tests, and deployment automated but not functional tests.  It appears that the SoapUI developers feel the same way and have provided some help in this area. There are two ways to run SoapUI functional tests automatically: integrate with JUnit or use the SoapUI “testrunner” shell script.   Unfortunately, the official documentation on both is sparse (see the hyperlinks in the previous sentence)  and not helpful beyond a bare minimum of support.

Note: All mention of SoapUI in this entry is related to version 3.5.1.

Integrating with JUnit

The documentation for how to integrate with JUnit is awful.  It barely provides the necessary information for using JUnit and only offers part of the story.  If you really want to know how to run your SoapUI tests from JUnit you’re mostly stuck with using the SoapUI JavaDocs and your own experimentation (Intellisense is your friend here).  However, the one advantage that this approach has over using the testrunner script is that you can have more control over how and when tests are executed. If you need that control but want to automate the process of running the test cases, this is the way to do it. But, if you just need to automate the test case runs without control, the command line test runner is preferable. Here is some help with the JUnit method.

To run SoapUI tests from JUnit tests you have to use the SoapUI API.  This means that the SoapUI libraries need to be in your Classpath.  The main archive needed is the bin\soapui-3.5.1.jar file.  If they’re not already in your Classpath, you may also need some of the JARs in the lib directory.  At a minimum you will need the xmlpublic-2.4.0.jar because it contains the XMLException class that is thrown by some of the classes in the SoapUI API.

When you’ve got your Classpath setup you can use the SoapUI API in your JUnit tests.  There are several ways to run SoapUI functional tests with the SoapUI API.  You can just run all the test cases in all the test suites in a SoapUI project without control over the order (you choose whether to run them synchronously or asynchronously), run a whole test suite, or run individual test cases within a test suite.  For the second and third options, you can either blindly iterate over test suites and test cases or you can run test suites or test cases by name.  Which way you run the tests is based on how much control you need over the order of test execution.  When running a test suite you have control over whether the test cases in the suite are run synchronously or asynchronously.  The options that you have available reflect what you can do in SoapUI’s user interface.  The problem is digging through the JavaDocs to find out which methods on what objects to call.

Here is an example of a JUnit test that runs all the test cases in a SoapUI project in two different ways.  The lessControl method just runs every test case in the project in the simplest way whereas the fullControl method iterates over the test suites and the test cases in the project.

package com.omig.soapUISamples;

import java.util.List;
import org.junit.Test;
import com.eviware.soapui.impl.wsdl.WsdlProject;
import com.eviware.soapui.model.support.PropertiesMap;
import com.eviware.soapui.model.testsuite.TestCase;
import com.eviware.soapui.model.testsuite.TestRunner;
import com.eviware.soapui.model.testsuite.TestSuite;
import com.eviware.soapui.model.testsuite.TestRunner.Status;
import com.eviware.soapui.tools.SoapUITestCaseRunner;
import static org.junit.Assert.*;

public class SoapUITest {

@Test
public void lessControl() throws Exception {
SoapUITestCaseRunner runner = new SoapUITestCaseRunner();
runner.setProjectFile(“SoapUIWorkspace/Sample-soapui-project.xml”);
runner.setPrintReport(true);  //Outputs a small table to stdout of test results.
runner.run();
}

@Test
public void fullControl() throws Exception {
WsdlProject project = new WsdlProject(“SoapUIWorkspace/Sample-soapui-project.xml”);
List<TestSuite> testSuites = project.getTestSuiteList();
for( TestSuite suite : testSuites ) {
List<TestCase> testCases = suite.getTestCaseList();
for( TestCase testCase : testCases ) {
System.out.println(“Running SoapUI test [" + testCase.getName() + "]“);
TestRunner runner2 = testCase.run(new PropertiesMap(), false);
assertEquals(Status.FINISHED, runner2.getStatus());
}
}
}
}

With the code above you’ve got a hint at how run SoapUI functional tests from a JUnit test.  To actually make this automatic you’ve got (at least) one more step.  In my situation we use Ant to script our software builds.  We then use Hudson to schedule the builds and make it all run automatically.  There are a lot of other ways you could do this but I’ll only show the Ant way here.  Any build tool that has support for JUnit will suffice.

Below is an example of an Ant task that runs the JUnit tests that are executing the SoapUI tests.  The JUnit output is being generated in XML format and sent to a folder named soapui-test-reports.

<path id="classpath">
    <!-- The path to the lib folder in the SoapUI installation.-->
    <fileset dir="${env.SOAPUI.LIB}"/>
    <!-- The path to the bin folder in the SoapUI installation.-->
    <fileset dir="${env.SOAPUI.BIN}"/>
</path>

<path id="soapui-workspace">
    <fileset dir="SoapUIWorkspace">
      <include name="**/*.xml"/>
    </fileset>
</path>

<target name="soapui-tests" depends="compile">
    <mkdir dir="../soapui-test-reports" />
    <junit printsummary="withOutAndErr"
           showoutput="true"
           haltonerror="on"
           haltonfailure="on"
           filtertrace="off"
           fork="no"
           forkmode="once">
      <classpath refid="classpath" />
      <classpath refid="soapui-workspace" />
      <formatter type="xml" />
      <batchtest todir="../soapui-test-reports">
        <fileset dir="tests">
                <include name="**/*SoapUI*.java"/>
        </fileset>
      </batchtest>
    </junit>
</target>

Testrunner Script

The documentation for the testrunner script is much clearer than for integrating with JUnit.  Integrating the testrunner.bat file with your build process is simple and straight-forward. An example of an Ant task to do this is shown below.  See SoapUI’s documentation for an explanation of the command line switches.

<target name="soapui-tests-cmdline" depends="compile">
        <exec executable="${env.SOAPUI.BIN}/testrunner.bat"
                  failonerror="yes"
                  failifexecutionfails="yes">
                <arg value="-rjaf"/>
                <arg path="../soapui-test-reports"/>
                <arg path="SoapUIWorkspace/Insured-Access-soapui-project.xml"/>
        </exec>
</target>

Which Method Is Best?

In my experience so far the command line testrunner tool is the better method for several reasons:

  1. There’s less work because you do not have to write a JUnit test suite/case to run the SoapUI tests.
  2. It creates a less fragile environment because writing a JUnit test to run the SoapUI tests makes it very difficult not to hard code the path to the SoapUI project file. You don’t have to worry about this with the command line because the input to script is parametrized.
  3. The command line test runner can output “correct” JUnit output. The JUnit reports generated by the command line test runner split results by test suite then test case. This is ideal when the results are viewed graphically because you get a full count of actual test cases run and you can drill-down to the individual test cases to see its run status*.
  4. To integrate with JUnit you have to use the SoapUI API, which is not documented well. There are Javadocs, but actual documentation is sparse in them.

* As far as I can tell it is difficult or impossible to do this by integrating with JUnit manually. To do so you would have to somehow generate one JUnit test for each test case in all the test suites. The SoapUI API allows you to easily iterate over the test suites and test cases, but there is no way in Java to dynamically generate methods for each test case.

About these ads

From → Uncategorized

7 Comments
  1. iggymacd permalink

    Hi,

    Great article! I have been working at this same approach for a few days now. In response to your footnote, I believe this works;

    SoapUITestCaseRunner runner = new SoapUITestCaseRunner();
    String projectFile = “../TddDemoWar/src/test/resources/DemoWebServiceTesting-soapui-project.xml”;
    String exportFolder = “../TddDemoWar/target/TddDemoWar/doc/report/function”;
    String[] options = { “-r”, “-A”, “-i”, “-f” + exportFolder, “-j”,
    projectFile };
    runner.runFromCommandLine(options);

    Cheers,

    Allan

    • thewonggei permalink

      Thanks for the comment, I did not see that runFromCommandLine method in the API. However, that does not solve the problem I posed in the footnote. The problem there is related to finding a way to actually generate a JUnit test case for each SoapUI test case. The result being the combination of fine-grained control over the execution of the SoapUI test suites and a complete JUnit test report. So far, it seems the only way to get a really useful JUnit test report, as you helped to point out, is to run the SoapUI tests from the command line.

  2. not entirely true this “As far as I can tell it is difficult or impossible to do this by integrating with JUnit manually. To do so you would have to somehow generate one JUnit test for each test case in all the test suites. The SoapUI API allows you to easily iterate over the test suites and test cases, but there is no way in Java to dynamically generate methods for each test case.”

    take alook here:
    http://barsand.wordpress.com/2011/09/24/running-soapui-parameterized-junit-tests-in-parallel/

    You can parametrize your testcases and iterate through the whole set if you want to run one by one.

  3. Hi,After writing the same code in JUnit Eclipse IDE- I am getting these error after running the code

    java.lang.NoClassDefFoundError: org/apache/commons/cli/CommandLineParser
    at com.omig.soapUISamples.Sample.testRunner(Sample.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
    Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.CommandLineParser
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    … 24 more

    Can anybody suggest me. How I can proceed further..?

    Thanks In Advance
    Pavan
    pavan.agadi@gmail.com

    • thewonggei permalink

      On first glance at this stack trace, it looks like the soapUI JAR files are not on the classpath in your Eclipse project.

Trackbacks & Pingbacks

  1. CXF Contract and JUnit First | Ménage à Mvn Eclipse Weblogic
  2. StirTrek: Darkness Edition | Nick A. Watts - The Blog

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 276 other followers

%d bloggers like this: