8.0.x
XLT 8.0.0
Version 8 is a major release of XLT. It comes with many new features, but also some incompatible changes. See below for an overview of what’s in the box.
Due to API changes in XLT 8, you may need to migrate your test suites for use with this new version. Please read the Test Suite Migration section carefully.
Overview
Framework
HtmlUnit, a browser emulation for Java, is the heart of the XLT framework. It has been updated to version 3.11. As there are some changes in the HtmlUnit API, please refer to the Test Suite Migration section.
Almost all other third-party libraries have also been updated, including Selenium.
XLT provides extensions for Chrome and Firefox to record request and response details when testing with these browsers instead of HtmlUnit’s browser emulation. The Chrome extension has been updated to be compatible with Chrome Manifest v3, which will be mandatory in the near future. The Chrome extension has also been enhanced to track additional metrics known as Web Vitals. These are metrics that help evaluate a website’s performance in terms of delivering a great user experience. XLT stores these metrics along with all other measurements in the timers.csv files.
Classes implementing the Data interface had the functionality to write/read their values to/from the CSV format. This functionality has been moved out of these classes and into the XLT core to give us more flexibility in the implementation details and the format itself.
XLT ships with some basic validation classes HtmlEndTagValidator, ContentLengthValidator, and XHTMLValidator. Since these validators are resource-intensive, they should be called when testing the load test scripts, but maybe not during the load test. For this purpose, HtmlEndTagValidator and ContentLengthValidator can now be enabled or disabled with a property, just like XHTMLValidator. Also note that all three validators are now disabled by default.
Load Testing
XLT may limit the creation of result browsers for scenarios that often fail for the same reason. When checking whether a scenario has already failed, XLT now considers not only the current scenario name and the failure stack trace, but also the name of the current action. This is necessary to support actions with dynamic names, such as AddToCart-US and AddToCart-DE.
Load Test Report
If a load test scenario defines a function instead of a constant value for either users, arrival rate, or load factor, that function alone defines the shape of the load curve. Any ramp-up parameters specified for this scenario are ignored in this case. That’s why a ramp-up period is no longer displayed in the Load Profile section of the test report when load functions are present.
There is a new Web Vitals page in the load test report. If the load test was run with Chromium browsers, this page displays the scores for all supported Web Vitals for each action in your scenarios. The score value is the 75th percentile of all observations and is rated as good, needs improvement, and poor using the colors green, yellow, and red according to Web Vital-specific thresholds.
The Errors page lists some paths to result browser directories for each different error entry. The maximum number of paths is now limited to 10 by default, but this can be configured in the report generator settings. Also, the directory list is better sampled to ensure that the result browsers are from different agents and from different times.
Comparison Report
The comparison report is no longer one long page with different sections, but the different sections have been moved to their own pages. The navigation remains the same, but points to the new subpages.
Comparison reports now display additional data. Pxx percentile data is now available on all pages. We have also added a table of bandwidth values to the Requests page.
XLT Images
Xceptance provides ready-to-use public XLT images for AWS and Docker. The images are now based on Debian 12.x (“bookworm”). They include the latest XLT and JDK 17 and are tuned for load testing. For real browser load testing, they also include the latest Chromium and Firefox (ESR) browsers with matching web driver binaries, chromedriver and geckodriver.
If you are looking for XLT images for other clouds, visit XLT’s sister project, XLT Packer. This project not only contains scripts to install and set up the contents of an XLT image, but also provides Packer templates to actually create such images for the following clouds:
- AWS
- DigitalOcean
- Hetzner
Follow the instructions in the README file to create an XLT image for one of the supported clouds yourself.
Test Suite Migration
As mentioned above, XLT 8.x comes with breaking changes. To migrate your test suites, you must complete the following steps.
1. Fix Compile Errors
HtmlUnit 3.11
There is no official list of breaking API changes in HtmlUnit between 3.0.0 and 3.11.0. Therefore, the following list is a work in progress. We will add to it as we find incompatibilities or are told about them by our users.
- The method
HtmlForm.getLostChildren()has been removed without replacement. Rewrite your code to work without it.
XLT Framework
The API of XLT’s Data classes, such as RequestData or ActionData, has been changed. Normally, XLT takes care of creating, populating, and writing objects of these classes to the timers.csv files under the hood. So if you don’t use these classes directly, there is nothing for you to do here.
However, if you have created custom data classes, please follow these steps:
- Rename the
setupRemainingValues()method tosetRemainingValues()and change its visibility to public. - Rename the
addValues()method totoList()and change its visibility to public. - Delete the
getMinNoCSVElements()method. It is no longer needed.
When in doubt, study the implementation of XLT’s data classes to learn how to rewrite your code.
2. Review the Configuration of Your Test Suite
XLT 8 changes the default values of some test suite properties. See the <your-test-suite>/config/default.properties file.
com.xceptance.xlt.api.validators.XHTMLValidator.enabled = false
com.xceptance.xlt.api.validators.HtmlEndTagValidator.enabled = false
com.xceptance.xlt.api.validators.ContentLengthValidator.enabled = false
Please check if the new defaults are suitable for you and adjust the values if necessary.
3. Test Your Scenarios
Now it’s time to verify that everything still works as before. Start by running the test scenarios from your IDE. They should run and finish with the expected results. Then run a small load test. Check the results and the output in all log files.
4. Review the Report Generator Configuration
As mentioned above, the XLT report generator has learned some new features. You can configure them in the `
## The maximum number of result browser directories displayed for each listed
## error (10 by default).
com.xceptance.xlt.reportgenerator.errors.directoryLimitPerError = 10
## The chance to replace directories when the maximum number of directories is
## reached ([0..1], default: 0.1, i.e. 10%).
## This ensures that directories processed earlier might also be replaced with
## directories processed later, resulting in a better sampling of directories
## across both agents and load test duration in the report.
com.xceptance.xlt.reportgenerator.errors.directoryReplacementChance = 0.1
Again, check the defaults to see if they work for you, and change them if necessary.
AWS AMI Retirement Notice
Xceptance provides machine images for Amazon’s Elastic Compute Cloud with XLT installed and configured for use as load agents. However, these images will not be available indefinitely. We strongly recommend that you migrate to the latest version frequently.
As part of our regular cleanup process, we will retire AWS AMIs older than XLT version 7.3.x on May 1, 2024. If you need an older version, please build your own AMI (see here for more details) or copy the appropriate AMI to your own AMI registry in advance.
If you need to maintain a particular setup for an extended period of time, please contact Xceptance and consider a support contract.