This post was written by Roy Sheaffer from Performance Center R&D
The online screen was one of our main focal points when developing Performance Center 12.55. We wanted to add powerful new capabilities that would make Performance Center a more effective tool for performance testers worldwide. This blog discusses these new features:
Automatic Anomaly Detection
Before discussing this feature, we need to first understand what is meant by the term “anomaly”. One definition is something that deviates from what is standard, normal, or expected. Based on this explanation, how should we define what is standard, normal, or expected of a measurement?
In Performance Center 12.55, a continuous weighted statistical analysis is performed for each measurement during the running of a test. This analysis produces a ‘sleeve’, which is then considered the expected range of a measurement.
To create this sleeve, we start by calculating the mean and standard deviation of a measurement. This is done for a learning period of 30 points (90 seconds). Then, we transition to a weighted algorithm to continuously update the ‘weighted mean’ and ‘weighted standard deviation’.
This algorithm gives higher priority to more recent points, and to points that are within the sleeve. Such an algorithm is required to allow the sleeve to be flexible enough to change quickly with the measurement. The sleeve is then centered at the weighted mean, with a width of 6 weighted standard deviations (3 above, and 3 below).
The output of our algorithm looks something like this:
Now that we have the expected behavior, we still need to decide what constitutes an anomaly. Is the deviation of a single point from the sleeve enough to be considered an anomaly? Would you consider that the above example contains an anomaly?
In the case of the above example, our algorithm will not alert that an anomaly has occurred. This is because such small deviations are far too common to justify notification. To help demonstrate this point, let’s see another example:
This second example (from the ‘CPU Utilization’ graph) shows a clear anomaly. The CPU is initially between 0 and 20 percent. While there are several deviations from the sleeve, they are too small and brief to be considered significant. That is, until a sharp spike occurs, that shows the CPU exceeding 50 percent. (Note: The plot band indicates the area in which the anomaly occurred).
Let’s suppose that a user runs a long test, and after a couple of hours they open the online screen and discovers that some of the graphs had anomalies ,indicated by the following icon:
To see the anomalies, the user would have to use the ‘whole load test’ granularity. But in this granularity, the level of detail is very low, and a 30 second spike might not even be noticeable.
To solve this issue and offer additional clarity, we developed the history view.
The History View
For users to really benefit from the anomaly detection feature, it was necessary to create the history view. In this view, the user can inspect the data of any time range during the test. This is much more powerful than the granularity settings in Performance Center 12.53.
To open the history view, click the options icon for a graph and select History.
The control displays the whole load test granularity, and the chart displays the time range that is selected in the control. Notice how you can immediately see when anomalies occurred throughout the load test, as indicated by the plot bands in the control. Furthermore, focusing on them is easy and intuitive and only requires changing the selected time range.
The selection can be changed by:
- Clicking the selection area and dragging it. This method changes the time range while maintaining the granularity.
- Clicking the left/right border of the selection and dragging it. This method will make the selection larger/smaller, and will therefore change the granularity.
- Creating a new selection by clicking an area in the control that is not within the existing selection, moving the mouse, and then releasing the mouse button.
- Clicking an anomaly plot band. This will select a time range that contains the anomaly.
Note: The highest granularity of the data is three minutes. You can select an area smaller than three minutes, but this will only result in fewer points displayed, not in a higher granularity.
While the history view was created with the anomaly detection feature in mind, it is a useful feature in its own right. Even users that have no interest in our automatic anomaly detection can still benefit from the history view.
Currently, the existing granularity settings are not flexible. As previously mentioned, viewing the first minutes of a test that was started several hours ago would require setting the granularity level to the whole load test. However, the degree of detail at such a granularity is very low. With the history view, users can now view the first minutes of the test at high granularity.
To allow the user to view any time range of any measurement, it was necessary to create a new database. This database is created at the start of a run, and is filled with data points throughout the duration of the run.
Now that such a database exists, what else can we do with it? This leads us to our third and final feature.
The Offline Screen
The offline screen allows you to view the data of runs that have finished running.
The offline screen is very similar to the online screen—but it displays completed actions. All of the graph-related actions that can be performed in the online screen are also available in the offline screen. For example, merged graphs can be created and measurements can be selected as favorites. Note: merged graphs that were created and settings that were applied online will be available offline.
There are also several differences, due to the fact that the run that is being viewed has already finished. For example, the graphs in the offline screen automatically appear in the history view, and the runtime view is not available. And of course, none of the Vuser/load generator/Controller-related actions are available.
The online screen is comprised of the following parts:
- A summary section that provides general data about the run.
- A global control that allows setting the time range for all visible graphs.
- The graph area and graph tree (same as in the online screen).
It can be opened from the following locations:
Note: To avoid issues with disk space, there is a process that is responsible for deleting the databases that are created for the history view and the offline screen. This process is based on two parameters:
- The last modification date. This field is updated every time that a run is viewed in the offline screen. The default setting is 30 days.
- The size limitation placed on the folder in which the run databases are stored. The default size is 10 GB.
Both of these parameters can be configured in the Performance Center Administration site to meet your personal requirements.
You can read more about these features in the Performance Center Help:
So what are you waiting for… try these performance testing features for yourself today.