Test run considerations for using Machine learning capabilities

Before you configure a Performance Test or Schedule run to use Machine learning (ML) capabilities, you must first read the considerations that you need to take into account.

When you want HCL DevOps Test Hub (Test Hub) to analyze Performance tests or Schedules by using ML capabilities, you must ensure that Performance tests or Schedules conform to the following criteria:
  • Must contain HTML tests.
  • Must contain stress or performance user profiles.
  • Must contain the following minimum stress test requirements:
    • 20 users or more.
    • A minimum of three different user groups and each group has an activity of more than two minutes.
    • Stages that have a reasonably high number of samples within each stage.
    • Stages do not have an increasing number of users.
    • Tests run for longer durations so that a trend can be observed.
You can refer to the following table to view the criteria for analysis of the parameters:
Parameter analyzed Criteria for analysis
Response Time Lock-Step Pattern The analyzer identifies the Response Time Lock-Step Pattern parameter in the overall page response time observed against the user count based on the following criteria:
  • Tests have a minimum of 20 users.
  • Tests run for longer durations for a trend to be observed.
  • Tests contain a minimum of three different user groups and each group has an activity of more than two minutes.
Response Time Standard Deviation Pattern The analyzer attempts to detect the response time of pages that are more than thrice the value of the standard deviation calculated for the page response time as the Response Time Standard Deviation Pattern parameter.
Throughput Drop Pattern The analyzer attempts to detect sudden drops in network throughput as the Throughput Drop Pattern parameter that is based on the following criteria:
Note: Sudden drops in throughput might be related to the performance tool itself, issues with network connectivity, or issues with signal-scalability of the system under test.
  • Tests have a minimum of 20 users.
  • Tests have stages with a reasonably high number of samples within a stage.
  • The length and intensity of the throughput drops that are less than 20% of the median observed within a stage are considered by the analyzer. When no throughput is received within 60 seconds, such drops are considered in the analysis. Short shark-tooth patterns in drops are not considered by the analyzer.
  • Parts of the stage with increasing users are not considered for analysis.

    For example, when the test adds one user every minute and the user range is between one user to 500 users, the Throughput Drop Pattern parameter is not analyzed.
Performance Agent Health The analyzer inspects the health of agents that are participating in a performance execution and identifies potential configuration issues based on the following criteria:
  • Minimum requirements
    • At least 10 users
    • At least 10-minute run time
    • Requires schedule with at least 1 static agent (Performance Agent Health insight is not generated if you are running the test on local host)
  • Blocking threads
    • Considered as Blocking threads when 90% of available threads are used for 20% of the run time
  • Pegged CPU
    • Considered as Pegged CPU when 90% of CPU is utilized for 10% of the run time
  • Pegged CPU for interval
    • Considered as Pegged CPU for interval when there are intervals of high CPU utilization greater than 95%

      For example, above 95% of CPU utilized for 50% of a 10-minute interval or 98% of CPU utilized over a shorter interval.

    • Considered as Pegged CPU for interval when there are spikes in CPU utilization during long test runs

The agents that are not configured properly can distort performance assessments of the system under test.

The analyzer can provide multiple insight types. The insight recommendation texts are parametrized and they provide specific recommendations such as identifying the agents with issues. You can view the following insights:
  • Pegged CPU for run insight
  • Pegged CPU for interval insight
  • Blocked thread insight