Part 3 of Agile testing of JIRA plugins: System tests - codecentric AG Blog

:

After unit tests and wired tests system tests are another test type. In this blog post we  – i.e. Thomas Strecker and I –  take a look at two test types which test the complete or almost complete application: tests which target the WebUI and tests which target the REST-API.

Testing the WebUI

Atlassian provides a number of tools in order to develop end-to-end tests for their products. Foremost among them are the TestKit, which enables the creation of a “backdoor” to perform administrative tasks and test data setup, and page objects for every application. While TestKit is not mandatory, its use is recommended, even if only to reset the test instance to a defined state.

Another tool for developing end-to-end tests is FuncTest. The main difference between TestKit and FuncTest lies in TestKit using REST-calls to perform administrative tasks, while FuncTest uses Selenium.

Setup

A typical setup of the dependencies, therefore, can be the following:

<dependency>
  <groupId>com.atlassian.jira.tests</groupId>
  <artifactId>jira-testkit-client</artifactId>
  <version>${testkit.version}</version>
  <scope>test</scope>
</dependency>
 
<dependency>
  <groupId>com.atlassian.jira</groupId>
  <artifactId>atlassian-jira-pageobjects</artifactId>
  <version>${jira.version}</version>
  <scope>test</scope>
  <exclusions>
    <!-- excluded due to clash with other SLF implementation -->
    <exclusion>
      <artifactId>slf4j-simple</artifactId>
      <groupId>org.slf4j</groupId>
    </exclusion>
  </exclusions>
</dependency>

<dependency> <groupId>com.atlassian.jira.tests</groupId> <artifactId>jira-testkit-client</artifactId> <version>${testkit.version}</version> <scope>test</scope> </dependency><dependency> <groupId>com.atlassian.jira</groupId> <artifactId>atlassian-jira-pageobjects</artifactId> <version>${jira.version}</version> <scope>test</scope> <exclusions> <!-- excluded due to clash with other SLF implementation --> <exclusion> <artifactId>slf4j-simple</artifactId> <groupId>org.slf4j</groupId> </exclusion> </exclusions> </dependency>

If you want to execute the tests on a 64bit Linux system like an CI server on an AWS EC2 instance, you may run into problems, however. The reason is that the Selenium version which comes as a dependency, uses Firefox 12. This firefox however has a problem on 64bit Linux systems.  The solution for this is to use a newer version of Selenium and the corresponding Atlassian browsers dependency:

<!-- the following dependencies are needed only for running on 64bit Linux, 
     since the default Firefox 12 has problems -->
<dependency>
  <groupId>com.atlassian.browsers</groupId>
  <artifactId>atlassian-browsers-auto</artifactId>
  <version>2.3.2</version>
  <scope>test</scope>
</dependency>
 
<dependency>
  <groupId>org.seleniumhq.selenium</groupId>
  <artifactId>selenium-java</artifactId>
  <version>2.25.0</version>
  <scope>test</scope>
</dependency>

<!-- the following dependencies are needed only for running on 64bit Linux, since the default Firefox 12 has problems --> <dependency> <groupId>com.atlassian.browsers</groupId> <artifactId>atlassian-browsers-auto</artifactId> <version>2.3.2</version> <scope>test</scope> </dependency><dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <version>2.25.0</version> <scope>test</scope> </dependency>

The final setup activity consists of activating the TestKit Plugin inside the Jira instance. This can be done by setting a property in the POM or by adding it as a plugin artifact to the JIRA build plugin’s configuration.

<plugins>com.atlassian.jira.tests:jira-testkit-plugin:${testkit.version}</plugins>

<plugins>com.atlassian.jira.tests:jira-testkit-plugin:${testkit.version}</plugins>

<pluginArtifact>
  <groupId>com.atlassian.jira.tests</groupId>
  <artifactId>jira-testkit-plugin</artifactId>
  <version>${testkit.version}</version>
</pluginArtifact>

<pluginArtifact> <groupId>com.atlassian.jira.tests</groupId> <artifactId>jira-testkit-plugin</artifactId> <version>${testkit.version}</version> </pluginArtifact>

Writing the test

The JIRA page objects artifact provides several means to effectively implement UI tests.

First of all it provides an abstract super class (AbstractJiraPage) for page objects. This class provides a skeleton for page objects. When extending this class, the methods getUrl() and isAt() must be implemented: getUrl() provides the URL to navigate to in order to open the page this object represents, while isAt() checks, whether the page is properly loaded (e.g. checking a given element is visible).

In addition, the class provides an instance of a PageBinder, which is a facility to bind the currently loaded page to a given page object, including wiring the page elements.

package pages;
 
import com.atlassian.jira.pageobjects.pages.AbstractJiraPage;
import com.atlassian.pageobjects.elements.ElementBy;
import com.atlassian.pageobjects.elements.PageElement;
import com.atlassian.pageobjects.elements.query.TimedCondition;
 
public class FooBarPage extends AbstractJiraPage {
  @ElementBy(id ="save") private PageElement saveButton;
  @ElementBy(id ="some-input") private PageElement someInput; 
 
  @Override
  public String getUrl() {
    return"/secure/admin/foo-bar.jspa";
  }
 
  @Override
  public TimedCondition isAt() {
    return someInput.timed().isVisible();
  }
 
  public FooBarPage save() {
    saveButton.click();
    return pageBinder.bind(FooBarPage.class);
  }
 
  public void setSomeInput(String input) {
    this.someInput.type(input);
  }
 
  public String getSomeInput() {
    return someInput.getValue();
  }
}

package pages;import com.atlassian.jira.pageobjects.pages.AbstractJiraPage; import com.atlassian.pageobjects.elements.ElementBy; import com.atlassian.pageobjects.elements.PageElement; import com.atlassian.pageobjects.elements.query.TimedCondition;public class FooBarPage extends AbstractJiraPage { @ElementBy(id ="save") private PageElement saveButton; @ElementBy(id ="some-input") private PageElement someInput; @Override public String getUrl() { return"/secure/admin/foo-bar.jspa"; }@Override public TimedCondition isAt() { return someInput.timed().isVisible(); }public FooBarPage save() { saveButton.click(); return pageBinder.bind(FooBarPage.class); }public void setSomeInput(String input) { this.someInput.type(input); }public String getSomeInput() { return someInput.getValue(); } }

A test class for testing the UI must be placed inside a package starting with it.*, since it is an integration test and requires a running JIRA instance.

package it.foo.bar;
 
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;
import org.junit.Test;
import pages.FooBarPage;
import com.atlassian.jira.functest.framework.FuncTestCase;
import com.atlassian.jira.pageobjects.JiraTestedProduct;
import com.atlassian.jira.pageobjects.config.EnvironmentBasedProductInstance;
import com.atlassian.jira.testkit.client.Backdoor;
import com.atlassian.jira.testkit.client.util.TestKitLocalEnvironmentData;
 
public class FooBarPageTest extends FuncTestCase {
  // the setupUpTest() method set this
  private JiraTestedProduct jira;
 
  @Override
  protected void setUpTest() {
    super.setUpTest();
    Backdoor backdoor = new Backdoor(new TestKitLocalEnvironmentData());
    backdoor.restoreBlankInstance();
    jira = new JiraTestedProduct(new EnvironmentBasedProductInstance());
  }
 
  @Test
  public void test_that_save_works() {
    FooBarPage page = jira.gotoLoginPage().loginAsSysAdmin(FooBarPage.class);
 
    page.setSomeInput("my input");
    page.save();
 
    jira.gotoHomePage();
    page = jira.goTo(FooBarPage.class)
    assertThat(page.getSomeInput(),is("my input"));
  }
}

package it.foo.bar;import static org.hamcrest.CoreMatchers.is; import static org.junit.Assert.assertThat; import org.junit.Test; import pages.FooBarPage; import com.atlassian.jira.functest.framework.FuncTestCase; import com.atlassian.jira.pageobjects.JiraTestedProduct; import com.atlassian.jira.pageobjects.config.EnvironmentBasedProductInstance; import com.atlassian.jira.testkit.client.Backdoor; import com.atlassian.jira.testkit.client.util.TestKitLocalEnvironmentData;public class FooBarPageTest extends FuncTestCase { // the setupUpTest() method set this private JiraTestedProduct jira;@Override protected void setUpTest() { super.setUpTest(); Backdoor backdoor = new Backdoor(new TestKitLocalEnvironmentData()); backdoor.restoreBlankInstance(); jira = new JiraTestedProduct(new EnvironmentBasedProductInstance()); }@Test public void test_that_save_works() { FooBarPage page = jira.gotoLoginPage().loginAsSysAdmin(FooBarPage.class);page.setSomeInput("my input"); page.save();jira.gotoHomePage(); page = jira.goTo(FooBarPage.class) assertThat(page.getSomeInput(),is("my input")); } }

Running the test

The frontend tests are executed as integration tests. Nothing special in this regard. There is however a small surprise: when using FuncTestCase as the super class an empty directory at src/test/xml is required. An alternative approach (which we recommend) is to not use FuncTestCase at all (the test extends java.lang.Object) and replace the setUpTest-method from above with:

@Before
public void setUp() {
  jira = TestedProductFactory.create(JiraTestedProduct.class);
  Backdoor backdoor = new Backdoor(
    new ProductInstanceBasedEnvironmentData(jira.getProductInstance()));
  backdoor.restoreBlankInstance();
}

@Before public void setUp() { jira = TestedProductFactory.create(JiraTestedProduct.class); Backdoor backdoor = new Backdoor( new ProductInstanceBasedEnvironmentData(jira.getProductInstance())); backdoor.restoreBlankInstance(); }

REST-API Tests

While the WebUI tests include the Java server and the Javascript client code this test type is known for its higher effort in the development and maintenance in addition to the longer runtime compared to other test types. One alternative for testing the complete server part are tests which target the server API. In the case of JIRA this is a REST-API.

It turns out that there is not much special about such REST-API tests in the context of a JIRA plugin. These tests are integration tests (meaning they are placed in the it.* package). A framework for developing REST tests has to be chosen. From our experience we can for example recommend REST Assured. There are only two items worth mentioning:

  1. The system property “baseurl” is set if the test is executed during the integration-test phase of Maven. It is not set, however, if the test is started from the IDE. Therefore, in such a case a fallback like “http://localhost:2990/jira” should be used.
  2. A preemptive authentication scheme should be used since JIRA does not send the HTTP status code of 401 (Unauthorized) but 200 (Ok) when authentication is required. This behavior is the reason why the automatic retry with authentication (as usually done by the HTTP client) does not work.

So one possible code part could be:

  String urlPrefix = System.getProperty("baseurl", "http://localhost:2990/jira");
  RestAssured.baseURI = urlPrefix + urlSuffix;
 
  PreemptiveBasicAuthScheme authScheme = new PreemptiveBasicAuthScheme();
  authScheme.setUserName(username);
  authScheme.setPassword(password);
  RestAssured.authentication = authScheme;

String urlPrefix = System.getProperty("baseurl", "http://localhost:2990/jira"); RestAssured.baseURI = urlPrefix + urlSuffix;PreemptiveBasicAuthScheme authScheme = new PreemptiveBasicAuthScheme(); authScheme.setUserName(username); authScheme.setPassword(password); RestAssured.authentication = authScheme;

Summary

The implementation of Selenium based frontend tests require only slight JIRA specific adaptations and running these tests on the local computer is nothing special. This is however different for running these tests on a CI server.  We will cover this topic in the next post. The REST-API tests on the other side show no problems, are much faster to implement and run and are therefore recommended as the second test type after unit tests. If you have a lot of Javascript and/or have to cover a wide range of browsers and/or operating system you should consider also adding WebUI tests.

In the next post we will take a look at the special things needed to run the tests on the CI server.