Testing

The base project for testing is called automated-testing. Here all source code for the tests and test framework is placed.

Prerequisites

In order for the existing tests to run properly, some prerequisites are needed:

• Selenium server: We are using Selenium as a running server for our tests. The latest version can be downloaded from the official page: https://www.selenium.dev/downloads/.

• Java: Tests are written exclusively using Java 17. The version can be installed using https://computingforgeeks.com/install-oracle-java-openjdk-on-debian-linux/ as a reference for Debian-based systems.

• Geckodriver: Tests are written for the Firefox browser, for which the Geckodriver is needed. We are using version 0.35, which can be downloaded from https://github.com/mozilla/geckodriver/releases.

• VNU Jar: Some tests need the VNU jar to be available on the machine. This can be downloaded using https://www.npmjs.com/package/vnu-jar as a reference.

• DSA server: Some tests need a DSA server to be available on the machine. This is done by modifying the orchestrator.conf file from your configuration path to have tester as both username and password.

Setup

In order to run the tests properly, follow these steps:

1. Clone the automated-testing project.

2. Run cd ./automated-testing/qa.

3. Edit qa/src/resources/testsConfig.ini file to match your data:

   • Modify the LDAP-related values.

   • Find the VNU jar path on your device and paste it under the correct variable name.

   • Make sure that the folders for downloads and screenshots exist.

   • Do not modify the name of the variables, as the tests will not work otherwise.

4. Save the file modifications.

5. Run ./gradlew clean (if using a Linux-based system) or gradle clean (if using Windows).

6. Run the tests:

   • To run all the tests, execute ./gradlew test (this will take some time as the code base is large, but you will see the test progress).

   • To run specific tests, execute ./gradlew test --tests "...", where ... is the path to your test starting from qa/src/test/java. This can be: * A full subfolder (e.g., org.fd.tests.core.* to run all tests in the core folder). * A specific class of tests (e.g., org.fd.tests.core.LoginTest to run all tests from the LoginTest class). * A specific test (e.g., org.fd.tests.core.LoginTest.testGoodLogin to only run testGoodLogin from the LoginTest class).

7. After tests complete, you will see:

   • BUILD SUCCESSFUL (in green) if all tests passed.

   • BUILD FAILED (in red) if any test failed.

   • In both cases, the test results will display:

     • PASSED for successful tests.

     • FAILED for failed tests, along with a short message (for troubleshooting, see Troubleshooting).

Troubleshooting

For every failed test, we have implemented features in our framework to aid troubleshooting:

• If the test is running in a visual interface, we take a screenshot of the last visual screen of the driver (saved in the screenshots folder defined in qa/src/resources/testsConfig.ini).

• Logs are saved in a file in the same screenshots folder. These logs help track the workflow of the test and identify where and why it crashed.

Project Structure

All important modifiable parts of the code base are inside the qa folder:

• config folder: Defines lint rules in XML format, checked by the pipeline whenever code is pushed to GitLab.

• build.gradle file: Defines the needed plugins for this project.

• src/test folder: Contains all the test code:

  • resources folder: Stores additional files:

    • testsConfig.ini: The initial configuration for the tests, should be modified as stated in Setup.

    • ldifs folder: Contains LDIF files inserted into the LDAP server before the tests run.

    • General rule: All additional non-Java files required for tests should be placed here.

  • test/org/fd folder: Contains all test classes:

    • Utils.java: A class with static fields, mainly Java translations of testsConfig.ini variables.

    • LdapConnection.java: Defines LDAP-related actions like emptying the LDAP (emptyLdap()) and inserting an LDIF (insertLdif(String filename)).

    • Assertions.java: Contains methods for assertions, such as checking if a user is logged in (assertLoggedIn(String username)).

    • ScreenshotTestWatcher.java: Defines actions executed after a test completes, including distinguishing between failed, successful, and aborted tests. See Troubleshooting.

    • FusionDirectoryTestCase.java: The main test class template in FusionDirectory. It integrates LdapConnection, Assertions, and ScreenshotTestWatcher with Selenium to interact with FusionDirectory’s web interface.

  • tests folder: Contains explicit tests:

    • core folder: Tests verifying the core functionality of FusionDirectory.

    • plugins folder: Contains one subfolder for each FusionDirectory plugin that has tests.

    • tools folder: Contains tests for various tools (schema manager, plugin manager, configuration manager, migration manager, and orchestrator client):

      • Unlike other folders, these tests often require Unix console access rather than web interface interaction.

      • CommandLineTestCase.java: Defines methods for interacting with the command line:

        • executeCommandWithWait(String command): Used when no user input is required.

        • executeCommandWithoutWait(String command): Used when user input is required, handled through the returned Response object.

      • CommandLineTestWatcher.java: A smaller version of ScreenshotTestWatcher.java that only copies log files, as screenshots are not applicable.

    • installation folder: Contains a single test verifying the installation page functionality.

How to Write a Test

Before writing a test, determine where it should be placed by asking, What am I testing? If the answer is a plugin abc, place the test in the plugins/abc folder. Then:

1. Create a class in the appropriate folder, ensuring the name ends in Test.

2. In the class constructor, define the initLdifs array with required LDIF files, or leave it empty if none are needed.

3. If altering FusionDirectory-related concepts, revert changes before the test ends or use an @AfterEach annotated method to clean up automatically.

4. When fetching web interface elements, check FusionDirectoryTestCase.java for existing methods before writing new ones to maintain clean tests and ensure easy maintenance.

5. Annotate test methods with @Test (or similar annotations like @RepeatedTest(n) or @ParameterizedTest(methodName)) for Gradle compatibility.

Before pushing to GitLab, perform these checks:

• Run ./gradlew checkstyleTest and ./gradlew spotbugsTest and fix any issues.

• Run the tests in your virtual machine and ensure they pass.

• Run the tests in a Bullseye Docker image and an Ubuntu Docker image to verify cross-platform compatibility.

By following these steps, we minimize failed pipelines and expedite code merging, ensuring tests facilitate development rather than hinder it. Reliable tests are key to this process.