Running UI tests in an Azure DevOps Pipeline

After you create the tests in HCL DevOps Test UI (Test UI) for the application that you are testing, and after you install the Test Workbench extension in your organization, you can run the tests in Azure DevOps pipelines.

Before you begin

You must have completed the following tasks:

About this task

After you add the Test Workbench extension in your Azure DevOps organization, you can use an existing pipeline or create a new one to add Test UI test tasks. You can install an agent or use the one that you installed in your default agent pool. You can add the Test UI tests to your task for the agent job, configure the task, and then run the task in the Azure DevOps pipeline.

If you have created test cases under test plans in your Azure DevOps project, you can provide the details of the Azure DevOps URL, test plan, test case, and your personal access token (PAT) while you configure the test job in a pipeline so that you can view the results of the test run on your Test Plan dashboard.

Procedure

  1. Open your Organization page in Azure DevOps and perform the following steps:
    1. Click the project you want to use.
    2. Initialize the repository by performing the following steps:
      1. Click Repos from the left pane.
      2. Click Initialize from the Initialize with a README or gitignore section.
        Note: Select the Add a README check box if it is not selected.
    3. Click Pipelines from the left pane.
    4. Click Create Pipeline.
    5. Click Use the classic editor to create a pipeline without YAML.
    6. Verify the project, repository, and branch for manual and scheduled builds, and then click Continue.
    7. Click Empty job.
  2. Select Pipeline and complete the following steps:
    1. Change the name for the build pipeline if required.
    2. Select the Agent pool for your build pipeline.

      You can use the agent from the default agent pool or use the one you have installed.

    3. Select the Agent Specification for the agent if required.
  3. Add a task to the agent job by completing the following steps:
    1. Click the Add Task icon Add Task icon in Azure DevOps pipeline for the agent job.

      The Add tasks pane is displayed.

    2. Search for the HCL tasks defined in the Test Workbench extension.

      The tasks that you can select are displayed.

      HCL Task selection in Azure DevOps

      Depending on the type of test that you have created in Test UI, you can select the type of task.

      You must use the following table to identify the task you must select:
      Type of test Task to select
      • AFT suites
      • Web UI tests
      • Compound tests
      DevOps Test UI - UI Test Task
    3. Select the DevOps Test UI - UI Test Task option, and then click Add to add the task to the agent job.

      The selected task is added to the agent job and it is displayed with a warning that some settings require attention. You must configure the settings mentioned in Step 4.

      You can also remove the tasks that are not required in your job. Select the tasks in the list that you want to remove. You can then right-click the tasks, and click Remove selected task(s) to remove them.

  4. Configure the settings by performing the following steps:
    1. Select the task version from the list if required.
    2. Follow the action for the Web UI task by referring to the following table:
      Note: All the required fields are marked with an asterisk (*) in the UI.
      Note:
      You must provide the values for the following required fields:
      • Workspace Location
      • Project Name
      • Test Suite Name
      If you include these required parameters in a configuration file and use the Configuration File field to specify the complete file path, then these values are not required.
      Field Description Action
      Display name Displays the name of the selected task. Enter the name of the task.
      Testcase Type The type of test to execute. Select Web UI from the Testcase Type list.
      Import The complete path to the project folder that is cloned from a source control system or remote repository. Enter the complete path to the project folder that is cloned from a source control system or remote repository. When you use this option, you can specifiy a name for the new workspace to be created to run the imported test assets.
      IMShared Path The path to the IMShared folder on your local computer. Enter the complete path to the location of the HCLIMShared folder if it is not in the default path. For example, D:\HCL\HCLIMShared
      Note: If you do not enter any value, the default path of the HCLIMShared folder is considered for test run. The default path in the different operating systems is as follows:
      • Linux: /opt/HCL/HCLIMShared
      • Mac: /Application/HCL/HCLIMShared
      • Windows: C:\Program Files\HCL\HCLIMShared
      Workspace Location The complete path to the Eclipse workspace. Enter the complete path of the Eclipse workspace.
      Project Name The name of the project containing the test. Enter the name of the project containing the test.
      Test Suite Name The name of a test within the project to use. A test can be a Web UI test, compound test, or AFT suites. Enter the name of the test that you want to run.
      VM Arguments Java virtual machine arguments to pass in.
      Enter the Java virtual machine arguments.
      Note: You can add multiple virtual machine arguments files separated by a comma.
      Labels The option to add labels to the test results when the test run is complete. Enter the labels to be added to the test results when the test run is complete. You can add multiple labels to a test result separated by a comma.

      For example, label1, label2

      When you run test assets then the same labels are displayed on the UI Test Statistical Report in Test UI.

      If you have set Publish result after execution as Always or Prompt in the Test UI preferences (Window > Preferences > Test > DevOps Test Hub) and use the Labels option, then the Results page of DevOps Test Hub displays the same label for the specific test asset.

      Note:
      • When you run tests by using the double quotation marks ("") for the Labels field, then the labels in the test result do not include the double quotation marks.
      • To work around this problem, you must create a command-line config file, and then run the test by using the Configuration File field.
      • When you use the Configuration File field to run tests, then labels provided in the configuration file take precedence over the labels provided in the Labels field.
      Var File The complete path to the XML file that contains the variable name and value pairs. Enter the complete path to the location of the variable file.
      Dataset Override For a test, the default value is the dataset specified in the test editor. Use the Dataset Override option to replace dataset values during a test run. If a test is associated with a dataset, you can replace the dataset at run time while initiating the run from the command line.
      Note:

      You must use the Dataset Override option to replace the dataset values during a test run. You must ensure that both original and new datasets are in the same workspace and have the same column names. You must also include the path to the dataset.

      For example,

      /project_name/ds_path/ds_filename.csv:/project_name/ds_path/new_ds_filename.csv.

      You can swap multiple datasets that are saved in a different project by adding multiple paths to the dataset separated by a semicolon (;).

      For example,

      /project_name1/ds_path/ds_filename.csv:/project_name1/ds_path/new_ds_filename.csv;
      /project_name2/ds_path/ds_filename.csv:/project_name2/ds_path/new_ds_filename.csv
      Configuration File The complete path to a file that contains the parameters for a test run. Enter the complete path of the file that contains the parameters for a test run.
      Results The name of the results file. The default name of the result file is the test name with a time stamp appended. Enter a folder name that is relative to the project to store the test results. For example, -results folder/resultname.
      Overwrite Determines whether a results file with the same name is overwritten. The default value false indicates that the new results file is created. Set the value as true to overwrite the existing file and retain the same file name.
      Export Stats The complete path to export the report to the format specified in the Export stats format field. Enter the values separated by comma to export those values to the CSV format.
      Export stat report list The option to list the reports that you want to export in place of the default reports, or the reports that are selected under Preferences. Enter the list of report IDs separated by comma.
      Export stats html The option to view and analyze the results on a web browser without using the test workbench. You can provide the complete path to a directory to export web analytic results. If you run multiple tests, do not provide a value in this field.
      User Comments The option to add and display your comments in the report. Enter the text within double quotation mark to display it in the User Comments row of the report.
      Note:
      • When you run tests by using the double quotation marks ("") for the User Comments field, then the User Comments row of a report does not contain double quotation marks.
      • To work around this problem, you must create a command-line config file, and then run the test by using the Configuration File field.
      Protocol Input The option to customize the test run. Choose one of the following options to customize the test run:
      • Run a Web UI test in parallel on different browsers.

        For example, Protocol Input is all.available.targets.in.parallel=all or

        Protocol Input is all.available.targets.in.parallel=chrome,ff,ie

        Note: If you use the Protocol Input argument, you must not use the equivalent VM Args arguments:
        VM Args = -Dall.available.targets.in.parallel=all
        VM Args = -Dall.available.targets.in.parallel=browser1,browser2,browser3
      • Run a UI test in the background through an agent.

        You can use bg=true to run a UI test in the background. The test execution continues even when the screen is locked or the agent is disconnected.

      Export Report The option to export the unified report to other file formats such as PDF, XML, and HTML. Enter the following details in the text field:

      type=<report type>;format=<file type>;folder=<destination folder path>;filename=<name of the exported file>

      Export stats format The option to specify a format for the report that you want to export.

      Specify a format for the report that you want to export. You must use the Export stats parameter along with the Export stats format parameter.

      You must use at least one of the following formats:
      • simple.csv

      • full.csv

      • simple.json

      • full.json

      • csv

      • json

      For example,
      Export stats = <local_dir_path> 
      Export stats format = json

      You can add multiple formats for the report separated by a comma (,).

      If you want to export both the simple and full reports in a json or csv format, you can specify json or csv as the format in the field.

      The reports are saved to the location specified in the Export stats field.

      publish The option to publish test results to DevOps Test Hub. Specify the server URL and project name to publish test results to DevOps Test Hub.

      You must provide the URL and offline user token of the server in Window > Preferences > Test > DevOps Test Hub of Test UI before you use the publish parameter in the test script.

      Use the following arguments with the publish parameter:

      • To specify the project name, use any of the following formats:
        • serverURL #project.name=projectName&teamspace.name=name_of_the _teamspace

        • serverURL #project.name=projectName&teamspace.alias=name_of_the _teamspace_alias

        You must consider the following points while providing the values:
        • If the project name is not specified, then the value of the Project Name parameter is used.

        • If you have a project with the same name in different team spaces, then you can append either the &teamspace.name=name_of_the _teamspace or &teamspace.alias==name_of_the_teamspace_alias options.

          For example:
          Workspace Location = C:/Users/IBM/devopstest/workspace1
          Project Name = proj1
          Test Suite Name = Tests/testHttp.testsuite
          publish = https://localhost:5443#project.name=test&teamspace.name=ts1
          Where:
          • https://localhost:5443 is the URL of the server.

          • test is the name of the project.

          • ts1 is the name of the team space.

        • If the name of the project or team space contains a special character, then you must replace it with %<Hex_value_of_special_character>.

          For example, if the name of the team space is Initial Team Space, then you must provide it as Intial%20Team%20Space.

          Where, %20 is the hexadecimal value of the space character.

        • You must not include the double quotation marks ("") while providing the value for the Publish field.
      • To avoid publishing of reports, use no.

        You can use the no option if you do not want to publish test results after the run. This option is useful if the product preferences are set to publish the results, but you do not want to publish them.

        For example:
        Workspace Location = C:/Users/IBM/devopstest/workspace1
        Project Name = proj1
        Test Suite Name = Tests/testHttp.testsuite
        publish = no

      If you do not use the Config File field to run the tests, then the values provided in the Publish field always take precedence over the Results options set in the product preferences (Window > Preferences > Test > DevOps Test Hub > Results).

      The Reports information section on the Output window displays the names of the report along with its corresponding URLs in the following conditions:

      • When you configured the URL of DevOps Test Hub in preferences of Test UI (Window > Preferences > Test > DevOps Test Hub.

      • When you set Publish result after execution as Always or Prompt in the preferences of Test UI (Window > Preferences > Test > DevOps Test Hub > Results).

      publish_for The option to publish test results to DevOps Test Hub based on the completion status of the tests.

      Specify the option to publish test results to DevOps Test Hub based on the completion status of the tests. You must use the publish_for parameter along with the publish parameter.

      The following are the available options that you can use for the publish_for parameter:

      • ALL - This is the default option. You can use this option to publish test results for any text execution verdict.
      • PASS - You can use this option to publish test results for the tests that have passed.
      • FAIL - You can use this option to publish test results for the tests that have failed.
      • ERROR - You can use this option to publish test results for the tests that included errors.
      • INCONCLUSIVE - You can use this option to publish test results for the inconclusive tests.
      You can add multiple parameters separated by a comma.
      For example:
      Workspace Location = C:/Users/IBM/devopstest/workspace1
      Project Name = proj1
      Test Suite Name = Tests/testHttp.testsuite
      publish = https://localhost:5443#project.name=test&teamspace.name=ts1
      publish_for = FAIL,ERROR
      publishreports The option to publish test results in DevOps Test Hub. Specify the option to publish test results in DevOps Test Hub. The values that you can use with publishreports are as follows:
      • FT - This is an identifier for Functional Test Report. You can use this value to publish the unified report if it is available for the selected test. See Unified reports.
      • STATS - This is an identifier for Statistics Report. You can use this value to publish the web analytics report if it is available for the selected test. See UI Test Statistical report.
      • TESTLOG - This is an identifier for Test Log. You can use this value to publish the test log if it is available for the selected test. See Logs overview.

      You must use the publishreports parameter along with the publish parameter.

      For example:
      Workspace Location = C:/Users/IBM/devopstest/workspace1
      Project Name = proj1
      Test Suite Name = Tests/testHttp.testsuite
      publish = https://localhost:5443#project.name=test&teamspace.name=ts1
      publishreports = STATS, TESTLOG

      The values specified here override the values selected in Window > Preferences > Test > DevOps Test Hub > Results of Test UI.

      You can prefix the value with ! to publish the reports except for the specified one in the test script.

      For example,
      Workspace Location = C:/Users/IBM/devopstest/workspace1
      Project Name = proj1
      Test Suite Name = Tests/testHttp.testsuite
      publish = https://localhost:5443#project.name=test&teamspace.name=ts1
      publishreports = !TESTLOG

      All the reports except the TESTLOG report is published to DevOps Test Hub after executing the command.

      Azure DevOps Project URL

      The URL of the test project in the organization on the Azure DevOps server.

      Note: You must enter the details for this option if you have created test cases under Test plans in your Azure project and want to view the test results on the Test Plans Dashboard.
      Enter the URL of the test project in the organization on the Azure DevOps server in the following format:

      https://<host>/<orgname>/<projectName>

      You must use this option with the following options:
      • Azure DevOps PAT
      • Test Plan Name
      • Test Case Name
      Azure DevOps PAT

      Your personal access token (PAT) of the Azure server where the test project that contains the test plan is hosted. You can also add your token to a secret variable and specify the variable name in the following form:

      $(variable_name)
      Note: The token must have read and write access.
      Enter your personal access token.
      You must use this option with the following options:
      • Azure DevOps Project URL
      • Test Plan Name
      • Test Case Name
      Test Plan Name The name of the test plan in the Azure server. Enter the name of the test plan.
      You must use this option with the following options:
      • Azure DevOps Project URL
      • Azure DevOps PAT
      • Test Case Name
      Test Case Name The name of the test case in the Azure server. Enter the name of the test case.
      You must use this option with the following options:
      • Azure DevOps Project URL
      • Azure DevOps PAT
      • Test Plan Name
    3. Expand Control Options and configure the settings for your task if required.
    4. Expand Output Variables and configure the settings for your task if required.
  5. Select the following options:
    1. Click Save to save the configured settings for the task.
      Note: The task is not queued for a run.

      You can save the task to a build pipeline and opt to run the build at a later time.

    2. Click Save & queue to save the configurations and queue the run in the pipeline.

      The Run pipeline dialog is displayed.

  6. Complete the following steps:
    1. Enter a comment for the test in the Save comment field.
    2. Select the agent that you configured for the test from the Agent pool list.
    3. Select the agent specification from the Agent Specification list for the agent if required.
    4. Select the branch from the Branch/tag list.
    5. Add the variables and demands for the task run from the Advanced Options pane if required.
    6. Select the Enable system diagnostics check box for a detailed log view.
    7. Click Save and run.

      The pipeline summary page displays the progress of the job run.

Results

You have run the tests for the application you are testing, in the Azure DevOps pipeline.

What to do next

  • You can open the job to view the task logs from the pipeline summary page.
  • You can click Test Plans to view the test result if you specified the values for Azure DevOps Project URL, Azure DevOps PAT, Test Plan Name, and Test Case Name.
  • You must click the task to open the Task page to view the test results.

  • You can access the report URLs to view the test execution information at any point of time. The report URLs are the HCL DevOps Test Hub URLs where the reports are stored.

    In Test UI, if the DevOps Test Hub URL is configured in Window > Preferences > Test > DevOps Test Hub and Publish result after execution is set as Always in Window > Preferences > Test > DevOps Test Hub > Results, then the logs in the Task page also displays the names of the published report along with its corresponding URLs.