Agile testing of JIRA plugins - codecentric AG Blog

:

Atlassian’s development infrastructure is quite sophisticated and developers usually get answers to most of the questions. The situation is slightly different, however, when it comes to questions about agile (i.e. automated, developer-driven) testing.

With a series of articles we – i.e. Raimar Falke and I – want to introduce developers which are new to JIRA plugin development to JIRA plugin testing, i.e. choose the right test types for their plugin and show how this testing is executed in detail. This first post contains an introduction to the topic, an overview of the tests in the context of a JIRA plugin in general and unit tests in particular.

JIRA and Plugins

JIRA is an issue and project tracking software by Atlassian which provides a rich set of features and is extremely customizable. It is used worldwide by a large number of companies, organizations and project teams.

Plugins also known as Add-Ons are the way of extending JIRA even further. While Atlassian already hosts a variety of plugins in their marketplace, there might be situations where a tailor made plugin is the only solution.

Fortunately, Atlassian provides a SDK for developing extensions for all of their products, as well a host of documentation and a questions & answers area.

A central part of any software project – and the development of a plugin for JIRA is one – is the test of the system. While Atlassian provides documentation and examples for most test-related use cases it is not always obvious which technologies or methodologies can – or cannot – be employed, especially if the tests should integrate themselves as smooth as possible into the existing development workflow.

Relevant and related technologies and terminology

The Atlassian products in general – and JIRA in particular – employ a set of technologies or terms which may be ambiguous or infamiliar to the reader. Therefore, we will introduce them to the extent which we deem reasonable in the context of this series.

Maven

Maven is the build management tool for all Atlassian products and extensions. It is  capable of handling extremely modular projects including their dependencies, build processes as well as reportings and can easily integrate into an Continuous Integration (CI) server. Atlassian provides wrappers for many maven commands to ease typical development tasks (cf. Atlassian Maven commands)

OSGi

OSGi is a consortium and a specification for modular Java software systems. Atlassian, like Eclipse, uses an OSGi container as the foundation of its products and all plugins are basically OSGi bundles. Therefore, certain restrictions and best practises stemming from OSGi must be taken into account during development – and even more so during testing. If we speak about a container in the the text below we mean the OSGi container.

Host application

The application like JIRA or Confluence which acts as a host to the plugin under development.

Active Objects

Active Objects is an ORM layer into Atlassian products. Since it is the recommended way of accessing and storing data, it should be taken into consideration when testing.

FastDev

Testing (manually and automatically) of a plugin running inside the container (e.g. to test the user interface) is tedious, because it requires to launch the container, JIRA, install the plugin and execute the tests repeatedly. With JIRA taking about 3 minutes per start-up, this quickly adds up to a large part of the day, even if changes between runs are minor. FastDev (a plugin itself) is a way to detect changes to the source code of the plugin from inside of the running JIRA and allows to rebuild and reload the plugin including the changes without having to restart the whole application, thus dramatically improving turnaround times.

Atlassian Maven commands

The following gives an overview over Atlassian commands for typical development tasks and their corresponding maven commands.

Command Maven version Description/Comment
atlas-clean mvn clean Cleans up the project (i.e. deletes the target folder).
atlas-unit-test mvn test Build the project and execute unit tests.
atlas-integration-test mvn integration-test Build the project, execute unit tests, launch a JIRA instance, install plugin(s) and execute integration tests inside/against this instance.
atlas-run mvn jira:run Build the project, execute unit tests, launch a JIRA instance and install plugin(s). Useful to re-use a running instance for development, thus saving time for start-up and shutdown. Add a version switch if you require a specific version of JIRA instead of the latest.
atlas-debug mvn jira:debug Build the project, execute unit tests, launch a JIRA instance and install plugin(s). In addition to the run command, a port for remote debugging is enabled.
atlas-install-plugin mvn jira:install Install the plugin to a running JIRA instance. Requires that the plugin is already built.
atlas-remote-test mvn jira:remote-test Build the project, execute unit test, install the plugin(s) to a running JIRA instance and execute integration tests there.
atlas-update mvn amps:update Updates the SDK to a new version.

Infrastructure Setup

Atlassian products are basically Java (web-)applications, which are built with Maven. The standard installation of the Atlassian SDK comes with its own Maven installation, a custom settings.xml, a local repository and a set of shell scripts (the above mentioned Atlassian Maven commands) which facilitate the development process.

The JDK, however, is a prerequisite. Our experiments revealed, that JIRA version up to 6.2.6 will not start when a JDK 8 is used. Therefore, we recommend using JDK 7, since it eliminates an issue with type inference which you could run into with JDK 6. Even if not explicitly explained (but in most examples you will find it set thus), the source and byte code must be JDK 6 compliant.

While the project was conducted, the SDK’s latest version (4.2.20) was still bundling Maven 2.1.0 which does not work with some plugins we find rather useful, among them FindBugs (which requires Maven 2.2.1) and Sonar (which needs Maven 2.2.x).

There are at least two ways in which the development infrastructure can be set up to work with a newer version of Maven, however.

  • Use the environment variable ATLAS_MVN (as explained here)
  • The value of the variable must point to the executable of your Maven installation (e.g. mvn.bat on Windows). If present all atlas-* commands will use this Maven executable to execute the actual commands (instead of the bundled maven), thus effectively switching to the given Maven installation. The drawback of this approach is that you will still need to use the atlas-* commands which some tools do not support.
  • Copy the settings.xml that comes with the SDK to your Maven installation’s or user settings
  • This will solve a lot of problems, including compilation problems with FastDev. The main benefit is the ability to use “pure” Maven commands, such as “mvn clean” (instead of “atlas-clean”), which eases integration with other tools, e.g. they can be issued with the standard means of most IDEs, too. It should be noted, however, that any existing configuration must be merged, and subsequent updates from the SDK must be manually incorporated. Another drawback is that these changes affect also other projects which may be not JIRA plugin projects. One alternative here for a good separation are multiple Maven installations in different directories (one patched for JIRA plugin development and one unchanged for other projects) and the switch is done using the PATH variable of the shell.

There are limitations to the versions of Maven you can use, however. Trial-and-error revealed, that 3.1.* or 3.2.* versions do not work due to a change in the felix plugin’s API, which the Atlassian SDK requires; Maven 3.0.* versions are fine. This is also the version we recommend. An example error message could be:

[ERROR] Failed to execute goal com.atlassian.maven.plugins:maven-jira-plugin:4.2.20:copy-bundled-dependencies (default-copy-bundled-dependencies) on project test: Execution default-copy-bundled-dependencies of goal com.atlassian.maven.plugins:maven-jira-plugin:4.2.20:copy-bundled-dependencies failed: An API incompatibility was encountered while executing com.atlassian.maven.plugins:maven-jira-plugin:4.2.20:copy-bundled-dependencies: java.lang.NoSuchMethodError: org.apache.maven.execution.MavenSession.getRepositorySession()Lorg/sonatype/aether/RepositorySystemSession;
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>com.atlassian.maven.plugins:maven-jira-plugin:4.2.20
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/C:/user/.m2/repository/com/atlassian/maven/plugins/maven-jira-plugin/4.2.20/maven-jira-plugin-4.2.20.jar

[ERROR] Failed to execute goal com.atlassian.maven.plugins:maven-jira-plugin:4.2.20:copy-bundled-dependencies (default-copy-bundled-dependencies) on project test: Execution default-copy-bundled-dependencies of goal com.atlassian.maven.plugins:maven-jira-plugin:4.2.20:copy-bundled-dependencies failed: An API incompatibility was encountered while executing com.atlassian.maven.plugins:maven-jira-plugin:4.2.20:copy-bundled-dependencies: java.lang.NoSuchMethodError: org.apache.maven.execution.MavenSession.getRepositorySession()Lorg/sonatype/aether/RepositorySystemSession; [ERROR] ----------------------------------------------------- [ERROR] realm = plugin>com.atlassian.maven.plugins:maven-jira-plugin:4.2.20 [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy [ERROR] urls[0] = file:/C:/user/.m2/repository/com/atlassian/maven/plugins/maven-jira-plugin/4.2.20/maven-jira-plugin-4.2.20.jar

How the developer wants to test

There are two main ways to run tests: during development in the IDE and at the CI server. The first helps the developer in the red-green-refactor cycle and the second in ensuring that no other functionality got broken during development. While speed is important in both cases running tests from the IDE is interactive and therefore speed is king. In this scenario it is also important to be able to select which test class(es) or test method(s) to run. On the CI server it is important that the tests are stable (no flaky tests breaking the build, build agents are equal,…) and that they are reproducible meaning that the context (OS, other support software, …) is well defined and can be recreated. Another difference is that on the CI server the tests are executed in a batch.

Regarding what tests to write the test pyramid usually gives the advice that there should be three test types:

  1. Unit tests try to test the component under test (the test subject) in isolation. For this the interaction with dependencies are controlled by the test. This is usually achieved using mocks which model the interface and contract of the dependencies. There are multiple reasons for mocks: they allow fine control of the behaviour and make it easy to also create unusual situations. Also mocks allow decoupling from external resources like network, database or the file system which are slow to access or difficult to set up.
  2. Service tests or subcutaneous tests which act as an end-to-end test without the difficulties of the UI.
  3. UI tests also include the frontend code in the test.

In the case of a JIRA plugin usually JavaScript code in the browser interacts with the Java part of the plugin in the JIRA server via a REST API. Therefore, the service test would test the REST API. And the UI tests would in addition also include the HTML and JavaScript code.

Available Test in JIRA

The following table shows the test types for a JIRA plugin we have identified. One aspect which needs to be considered for each test type is where the test method is executed and where the test subject runs. Normally the test method is run in the original VM (created by the CI server or the IDE). However for JIRA plugins there is also a test type where the test method runs inside the host application. The same distinction can be made for the test subject. Think about a front end test with Selenium: the test method runs on a local VM but the test subject runs on the server in a different VM.

Test Type Test code runs in Test subject runs in
Unit tests original VM original VM
Integration tests original VM original VM
“Traditional Integration Tests” (Atlassian speak) original VM host application
“Wired Tests” (Atlassian speak) host application host application

Unit Test

For unit testing JIRA plugins it is recommended by Atlassian, although not really required, to place the tests inside the ut.* packages (“ut” stands for unit tests). This serves to distinguish them from integration tests (which will reside inside the it.* packages) or normal supporting classes (e.g. page objects and utilities).

As stated above, unit tests serve to test an isolated unit of the system. In order to be able to test units in isolation, it is necessary to either develop rather loosely coupled and independent units or make use of mock frameworks.

Dependencies

In order to create unit tests at least the following dependencies should be included. Among other things this brings a lot of mock-objects to work with and a dependency to mockito.

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.11</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.atlassian.jira</groupId>
    <artifactId>jira-tests</artifactId>
    <version>${jira.version}</version>
    <scope>provided</scope>
</dependency>

<dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> <scope>test</scope> </dependency> <dependency> <groupId>com.atlassian.jira</groupId> <artifactId>jira-tests</artifactId> <version>${jira.version}</version> <scope>provided</scope> </dependency>

Mocking

Unit tests can then create mocked objects in the usual way:

MutableIssue issue = mock(MutableIssue.class);
Project project = mock(Project.class);
when(issue.getProjectObject()).thenReturn(project);
when(issueManager.getIssueObject(1)).thenReturn(issue);

MutableIssue issue = mock(MutableIssue.class); Project project = mock(Project.class); when(issue.getProjectObject()).thenReturn(project); when(issueManager.getIssueObject(1)).thenReturn(issue);

A specialty of OSGi is the use of dependency injection through the constructor. As a result, most components in a JIRA plugin have a rather large number of constructor parameters. In order to test these components, all dependencies have to be mocked (FooBar is the component under test):

I18nHelper i18nHelper = mock(I18nHelper.class);
PermissionManager permissionManager = mock(PermissionManager.class);
IssueManager issueManager = mock(IssueManager.class);
FooBar foo = new FooBar(i18nHelper, permissionManager, issueManager);

I18nHelper i18nHelper = mock(I18nHelper.class); PermissionManager permissionManager = mock(PermissionManager.class); IssueManager issueManager = mock(IssueManager.class); FooBar foo = new FooBar(i18nHelper, permissionManager, issueManager);

An alternative to this type of dependency injection is the use of the ComponentAccessor. While this may seem to unclutter the component, it has some drawbacks, especially in the face of unit tests, when the system is not fully deployed and the ComponentAccessor will fail to provide the component because it is not initialized. A solution here is the use and initialization of a MockComponentWorker which will provide the ComponentAccessor with the requested components (note that the objects are identical to the previously created mocks):

new MockComponentWorker()
    .addMock(PermissionManager.class, permissionManager)
    .addMock(I18nHelper.class, i18nHelper)
    .addMock(IssueManager.class, issueManager).init();

new MockComponentWorker() .addMock(PermissionManager.class, permissionManager) .addMock(I18nHelper.class, i18nHelper) .addMock(IssueManager.class, issueManager).init();

We advise, however, to use constructor based dependency injection and not ComponentAccessor/MockComponentWorker because the constructor shows in a concentrated form the list of all dependencies. Otherwise you would have to search for all ComponentAccessor usages or use trial-and-error to get the correct MockComponentWorker call chain.

Testing Active Objects

In order to test persistent objects which rely on the Active Objects framework – we will call them repositories from now on – additional dependencies are required (note the use of a property in place of the version enabling synchronization of test and framework dependencies):

<dependency>
    <groupId>net.java.dev.activeobjects</groupId>
    <artifactId>activeobjects-test</artifactId>
    <version>${ao.version}</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.atlassian.activeobjects</groupId>
    <artifactId>activeobjects-test</artifactId>
    <version>${ao.version}</version>
    <scope>test</scope>
</dependency>

<dependency> <groupId>net.java.dev.activeobjects</groupId> <artifactId>activeobjects-test</artifactId> <version>${ao.version}</version> <scope>test</scope> </dependency> <dependency> <groupId>com.atlassian.activeobjects</groupId> <artifactId>activeobjects-test</artifactId> <version>${ao.version}</version> <scope>test</scope> </dependency>

The tests themselves are regular JUnit tests with additional annotations (see below for an example):

  1. Tests for Active Objects are run with a specific test runner.
  2. The runner must be instructed which (type of) database to use for the tests.
  3. A class for preparing the test database is required.

For the last point, an implementation of the DatabaseUpdater interface must be provided. According to the documentation, this updater is called once per class (or it is even reused across multiple class if the implementation is shared). In its update method it must tell the entity manager to migrate (prepare the database for) all relevant entity classes:

public class TestDatabaseUpdater implements DatabaseUpdater {
 
    @Override
    public void update(EntityManager entityManager) throws Exception {
        entityManager.migrate(Foo.class, Bar.class);
    }
}

public class TestDatabaseUpdater implements DatabaseUpdater {@Override public void update(EntityManager entityManager) throws Exception { entityManager.migrate(Foo.class, Bar.class); } }

For the database, a wide range of servers is supported, e.g. HSQL (in-memory and file storage), MySQL, Postgres or Derby.

By default every test is executed inside its own transaction, which is rolled back afterwards. This works, however,  only if the class under test (the repository) leaves the transaction handling to the container (as described in the second half of this document). If you follow the implementation pattern described in the first half of the referenced chapter, i.e. the repository takes control of the transactions, it is necessary to annotate each test with @NonTransactional. The following snippet shows a sample test class (which is using the database updater shown above):

@RunWith(ActiveObjectsJUnitRunner.class)
@Data(TestDatabaseUpdater.class)
@Jdbc(Hsql.class)
public class FooRepositoryTest {
 
    // gets injected by the ActiveObjectsJUnitRunner
    private EntityManager entityManager;
 
    // AO repository under test
    private FooRepository fooRepository;
 
    @Before
    public void setup() {
        this.fooRepository = new FooRepositoryImpl(new TestActiveObjects(entityManager));
    }
 
    @Test
    @NonTransactional
    public void test_that_saved_value_can_be_retrieved() {
        Foo foo = new Foo("test");
        this.fooRepository.save(foo);
        List<Foo> foos = this.fooRepository.findAll();
        assertThat(foos, hasItem(
            Matchers.<Foo> hasProperty("name", is("test"))));
    }
}

@RunWith(ActiveObjectsJUnitRunner.class) @Data(TestDatabaseUpdater.class) @Jdbc(Hsql.class) public class FooRepositoryTest {// gets injected by the ActiveObjectsJUnitRunner private EntityManager entityManager;// AO repository under test private FooRepository fooRepository;@Before public void setup() { this.fooRepository = new FooRepositoryImpl(new TestActiveObjects(entityManager)); }@Test @NonTransactional public void test_that_saved_value_can_be_retrieved() { Foo foo = new Foo("test"); this.fooRepository.save(foo); List<Foo> foos = this.fooRepository.findAll(); assertThat(foos, hasItem( Matchers.<Foo> hasProperty("name", is("test")))); } }

Running unit tests

Unit tests are normally run with the command “atlas-unit-test”. If the development environment is set up as described above, it is also possible to run the tests with the command “mvn test” or from inside an IDE with the unit test runner.

Summary

There are a few traps with the basic setup of an JIRA plugin project which we outlined above. Implementing and executing basic unit tests in contrast is quite straightforward. In the next post we take a look at “wired tests”: what are these and how can these help the developer.

Other parts of this series

Part 2 of Agile testing of JIRA plugins: Wired Tests

Part 3 of Agile testing of JIRA plugins: System tests