4.8.x

XLT 4.8.3

Load Testing

Similar scenario execution paths despite randomization (#2900)

In XLT 4.8.2, we fixed an issue in XltRandom that caused randomized execution paths and test data to be less random among virtual users than expected. Unfortunately, this fix introduced another issue that may seriously spoil any randomization in a test scenario. After a couple of iterations, a virtual user just chose from only two different seed values when reinitializing its XltRandom instance for the next iteration. As a consequence, the “randomness” created this way was the same in every other iteration. This new issue is fixed now.

XLT 4.8.2

Script Developer

Support Firefox 52 (#2884)

Script Developer has been made compatible with Firefox 52, which was released on March 7, 2017. Script Developer runs on Firefox 45/ESR up to 52 now.

Provide additional keyboard shortcuts (#2871)

Now almost all operations are available not only via context menu, but also via keyboard shortcut. See below for a list of the new shortcuts.

Display full package name in tooltip (#2892)

When hovering over a module call in the script editor, the tooltip now shows not only the module name, but also the full package name.

Resume Replay keyboard shortcut causes script to start anew (#2860, #2861)

When the script currently being replayed is paused (manually or when reaching a breakpoint), execution can be resumed via the toolbar button ▹ or the keyboard shortcut Alt+F5. While the former works as expected, the latter caused the script to start from the beginning. Fixed.

Incorrect determination of element style (#2869)

When Script Developer needs to determine the visibility of an element, the result was not always correct. This happened if style properties were overwritten with an !important value. Fixed.

Script explorer view not updated when en-/disabling test case (#2876)

When you enable or disable a script test case, the test case will be shown either regular or crossed-out, respectively. However, sometimes the script explorer’s tree view was not updated accordingly, especially when the corresponding panel is rather narrow. Fixed.

Load Testing

XLT AMIs should support setting a custom agent controller password (#2843)

Access to the agent controllers can be secured by defining a custom password at both the master and agent controller machines. Usually, this involves to log on each machine and change the password manually, or an automated version of that.

When using our public XLT images (tagged ‘XLT 4.8.2’ and above), setting the password can now be done automatically for you. To this end, you need to provide the password as part of the user data that will be passed to each machine instance during start-up. Previously, user data was solely used to specify custom host name mappings that were simply appended to the instance’s /etc/hosts file. In order to allow for other data as well, we redefined the format of user data to be JSON with the following (optional) fields:

{  
    “acPassword”: “<pairing password to connect agent and master controller>”,  
    “hostData”: “<data to add to ‘/etc/hosts’>”  
}

Note that in case the user data is not JSON, the data is treated as host name mappings and will be appended to /etc/hosts as before.

If you start XLT images via the AWS EC2 web console, provide an appropriate JSON string as user data. The image will then parse the data from the given JSON and apply it accordingly. In case you’re using ec2_admin to start your AMIs, you neither need to remember the format nor to prepare the JSON by yourself. Instead, you will be asked to enter password and host data separately, and the JSON is built for you in the background.

ec2_admin should provide a sub-net when starting instances (#2868)

Our ec2_admin utility has been enhanced to automatically specify a sub-net when starting machine instances, because VPN instances require such a sub-net. ec2_admin chooses the default sub-net, or the first sub-net found if no default is defined (which is the case for “older” AWS regions).

Mask passwords shown in the test report (#2865)

In case you need to share load test reports with others, you may not want to expose passwords and other sensitive information, but they appear in clear text in the configuration section of the report. That’s why the report generator can be configured to mask the values of certain properties, i.e. their values will be replaced with ******. Specify the properties to be masked using a regular expression in <xlt>/config/reportgenerator.properties, for example:

com.xceptance.xlt.reportgenerator.maskPropertiesRegex = (?i)(password|passphrase|login)

By default, any property with the substring password in the name will be masked.

Framework

NPE when invoking a mutation observer (#2859)

JavaScript code on a page may attach a mutation observer that listens for DOM mutations. When setting an attribute at an element, the mutation observer will be invoked, but sometimes this failed with a NullPointerException. Fixed in HtmlUnit.

Unknown host issues lead to negative time-to-first/last-byte values (#2873)

When running test scenarios with HtmlUnit or XltDriver, the time-to-first-byte and time-to-last-byte values recorded for a request could be negative. This happened only if the host name of the target server could not be resolved to an IP address. Fixed.

XltDriver does not obey an implicit wait time of 0 (#2854)

If the configured implicit wait timeout is 0, XltDriver did nevertheless sleep for one round (200 ms) before returning from an unsuccessful find operation. This made XltDriver-based test cases run a bit longer than necessary. Fixed.

XltChromeDriver/XltFirefoxDriver fails to fetch performance data (#2877)

If there is only one browser window open and the test case calls WebDriver.close() before - or even without - calling WebDriver.quit(), request and browser event data recorded during test execution was lost, either completely (Chrome) or partially (Firefox). Fixed.

Time-to-first/last-byte values are always 0 for Firefox (#2879)

When a test scenario uses XltFirefoxDriver, XLT records also the network timings for each request. However, the values for time-to-first-byte and time-to-last byte were always 0. Fixed.

Implausible request timestamps for Chrome (#2886)

When a test scenario uses XltChromeDriver, XLT records also request and browser event timings. However, these timings were not always plausible. Sometimes the timestamp of embedded resources was smaller than the one of the main request, or the timestamp was even 0. Fixed now.

FirefoxBinary argument ignored when constructing an XltFirefoxDriver (#2893)

When creating an XltFirefoxDriver object using the constructor

public XltFirefoxDriver(FirefoxBinary binary, FirefoxProfile
profile, Capabilities desiredCapabilities, Capabilities
requiredCapabilities)

the FirefoxBinary argument was not passed to the super class and was hence completely ignored. Fixed.

Similar scenario execution paths despite randomization (#2890)

The virtual users executing a certain test scenario chose seed values from a similar value range when initializing their private XltRandom instance. As a consequence, even if the scenario is heavily randomized, the resulting execution paths and test data could be the same. Fixed.

Empty result browser if strict MIME type checking enabled in Chrome (#2862)

When viewing XLT result browsers with Chrome, an empty result browser could be shown. This happened if strict MIME type checking was enabled in Chrome, preventing Chrome from loading the result browser data file. Fixed.

Frame documents not displayed in the result browser (#2878)

When XLT result browsers are served from a Web server, document files for (i)frame elements might be served using an incorrect content type, such as application/octet-stream instead of text/html. This causes the browser to open a download dialog when viewing pages with frames. In order to give the server a hint about the type of a frame document, XLT now stores these documents with an .html file extension.

XLT 4.8.1

XLT Test Automation Survey

Please take part in our XLT Test Automation Survey and help shape the future of XLT. It will only take about 10 to 15 minutes and you might win a prize. We appreciate your feedback!

XLT Test Automation Survey

This way to the survey: https://goo.gl/forms/vtKvIzIeEmzpepVE2

Please share this survey with others who are using XLT for test automation, are interested in using it, or would use it if it fulfilled certain criteria.

Thank you in advance,
Your Xceptance Product Team

Script Developer

Support Firefox 51 (#2849)

Script Developer has been made compatible with Firefox 51, which will be released on January 24, 2017. Script Developer runs on Firefox 45/ESR up to 51 now.

Number of matching elements does not correspond to locator (#2844)

While entering/modifying a locator expression in the Command Details dialog, the number of matching elements will be updated as you type. Due to a race condition, the number shown might not correspond to the locator in its final form, but an intermediate version only. Fixed.

Record mode can’t be stopped after switching browser tabs (#2845)

When you switch to another browser tab while recording a script, Script Developer goes into pause mode. In this mode, recording could not be stopped. Fixed.

Load Testing

AWS data centers in London and Canada (#2846)

Recently, Amazon opened two new data centers in London (eu-west-2 ) and Canada (ca-central-1 ). These new data centers are now fully supported in ec2_admin, your tool to manage your AWS EC2 machine instances.

ec2_admin in non-interactive mode fails to list agent controller URLs (#2855)

When you use ec2_admin in non-interactive mode to start EC2 machine instances, ec2_admin started the instances successfully, but failed when listing the corresponding agent controller URLs, emitting the error message “Failed: Format specifier ‘%s’”. Fixed now.

XLT Framework

Certain script commands take longer than necessary (#1734, #2850)

When replaying script test cases via the XLT framework, script commands that check for the absence of elements might run longer than expected. This happened only if an implicit wait timeout is set. For example, with an implicit wait timeout of 5 seconds the command assertNotElementPresent did not return before that timeout, although the element in question did not exist on the page right from the beginning.

The following commands were affected:

• assert/waitFor … NotElementPresent
• assert/waitFor … NotElementCount (only when count != 0)
• assert/waitFor … NotXpathCount (only when count != 0)
• assert/waitFor … ElementCount (only when count 0)
• assert/waitFor … XpathCount (only when count 0)

The reason for this misbehavior is that there is no way to check for the absence of an element other than trying to find it and expecting an empty result. Unfortunately, when finding elements the implicit wait timeout kicks in, causing the find method to wait for the element in question. However, why waiting for an element to be present when we expect it to be not present?! In these cases, the timeout is counterproductive and will now be ignored.

XLT Jenkins Plugin

The XLT plugin for Jenkins has been updated to support the new AWS data centers in London and Canada.

Posters Store

There is also a new version of the Posters store, our demo application shipped with XLT. It fixes a small issue that appeared when load-testing the store. The corresponding demo test project testsuite-posters has been updated accordingly.

XLT 4.8.0

Release Overview

XLT 4.8.0 is primarily a technology update release, but also comes with some new features. This section summarizes the highlights of this release.

First and foremost, XLT now ships with Selenium 3, the new version of the WebDriver library. All other core libraries have been updated as well. This also includes HtmlUnit for an improved browser emulation. Beginning with this release, XLT requires Java 8 to run.

The XLT framework also comes with some functional improvements. XLT now supports OperaDriver out of the box and can run FirefoxDriver in either the new Marionette mode using geckodriver or in the “old” legacy mode. When you drive Firefox via XltFirefoxDriver, you will get a much more detailed result browser now, with almost the same request and response details that you already know from XltDriver. Any values that you programmatically add to the newly introduced value log of a session are shown in the result browser as well. Furthermore, most of the XLT framework properties can now be configured not only globally, but also specifically for a certain test scenario.

For load testers, there is something in the box as well. If you use the AWS EC2 cloud a lot, you will be glad to hear that the new AWS data center in Ohio is now fully supported. The load test report has been tuned to become usable much faster, even with lengthy pages such as the Requests page.

Last but not least, the Poster Store demo application and the XLT Jenkins Plug-In have both been updated. If you ever wanted to load-test your WebDAV server, there is now a new demo test suite for that.

Sounds interesting? Continue reading to get all the details.

Framework

Selenium 3

Selenium 3 is now available and bundled with XLT. If you used solely the WebDriver API in the past, there are no changes between v2 and v3. Thus no need to adjust your test case code. See here for all the details.

Selenium 3 requires browser vendors to provide a browser-specific driver binary. You will know this approach already from Chrome and other browsers. The driver binary translates the browser-neutral WebDriver wire protocol into browser-specific actions. Firefox was the last browser without such a driver, but Mozilla has now delivered. Welcome, geckodriver!

See below for a list of links from which you can download the driver binary for your browser:

• Chrome/Chromium: chromedriver
• Edge: MicrosoftWebDriver.exe
• Firefox: geckodriver
• Internet Explorer: IEDriverServer.exe
• Opera: phantomjs

Since geckodriver is not complete yet and may have bugs, Selenium still supports the “old way” where a special WebDriver extension will be installed into Firefox (legacy mode). Give it a try in case you experience issues with geckodriver.

To switch between geckodriver mode and legacy mode, change the following setting as needed:

xlt.webDriver.firefox.legacyMode = true  
xlt.webDriver.firefox_clientperformance.legacyMode = true

But note that the legacy WebDriver Firefox extension is not signed by Mozilla, so you won’t be able to run your tests against Firefox 48 or higher as these versions run signed extensions only. Either downgrade your Firefox to version 47.0.1 or, even better, use the Extended Support Release of Firefox (Firefox 45/ESR).

OperaDriver supported

If you want to use the Opera browser for your WebDriver-based test cases, this is now possible in a very convenient way. Simply choose opera as the WebDriver to use in the test suite configuration:

xlt.webDriver = opera
#xlt.webDriver.opera.pathToDriverServer = /path/to/operadriver  
#xlt.webDriver.opera.pathToBrowser = /path/to/opera

Please make sure you also have the operadriver executable installed on your machine. See above for the download link.

In case you don’t want to use the default Opera installation, but a different one, or the location of the driver server is not in your PATH, you can configure alternative paths as needed.

Result browser with request/response details also for Firefox

We have improved our add-on for Firefox to provide not only timing data, but also other request and response details. The response body is the only piece not available yet. All this data is shown in the result browser the same way as you know it from XltDriver or XltChromeDriver. There is nothing special to do to get this data. Just make sure you use XltFirefoxDriver instead of FirefoxDriver when running tests with Firefox.

Timer data in timers.csv

When using XltFirefoxDriver or XltChromeDriver, request and browser event timings are gathered. Previously, this data was stored to separate timer-wd-<sessionid>.csv files, while transaction and action timings went into the timers.csv file. Now all data is in timers.csv.

Moreover, the recorded browser events and also their names were different between Chrome and Firefox. This has been unified across browsers. See below for the list of browser events and their (new) name:

• DomLoading
• DomInteractive
• DomComplete
• DomContentLoadedEventStart
• DomContentLoadedEventEnd
• LoadEventStart
• LoadEventEnd
• FirstPaint (only available for Chrome right now)

General value log in result browser

The XLT session now features a general value log. Any value you add to this log will be made available in the result browser. To view the value log, click the new link with the transaction’s name that is located above the action list.

Value Log in the Result Browser

This feature is primarily intended to aid in error analysis. The data in the result browser may help you to reconstruct and rerun a failed test case iteration without having to dig into log files, etc. Simply add any value of special interest and it will be available in the result browser. This is especially useful if your test case uses random or randomly chosen test parameters.

To address the problem with random test parameters, XLT adopts this feature to make the seed value of XltRandom - used for the current test iteration/session - available in the result browser. To rerun a test case with this seed value (i.e. with the “same randomness”), copy the seed value from the result browser, add the following line to your dev.properties file, and re-execute the test as usual from your favorite IDE:

com.xceptance.xlt.random.initValue = <copied seed value>

You may also add your own arbitrary values to the session’s value log. See below on how to get access to the value log and add values to it:

Map<String, Object> valueLog = Session.getCurrent().getValueLog();  
valueLog.put(“user”, randomEmail);  
valueLog.put(“password”, randomPassword);

Your data will be stored as simple name/value pairs. Even though the log accepts any Object as value, it still needs to be converted to a string for proper display in the result browser, so make sure that your value classes provide a sensible toString() method.

Note that in a load test the value log will be cleared automatically between two iterations of your test scenario.

Scenario-specific overrides of framework properties

The XLT framework provides a lot of settings to influence the behavior of your test cases. Up to now, you could configure them globally only, so the behavior was changed the same way for all test scenarios at once. While this makes sense most of the time, there are use cases where scenario-specific settings are advantageous. For example, if you mix different types of test cases in a load test (Web interface tests and API tests), you may only want the Web tests to simulate a “user think time” between actions. Such a scenario-specific configuration is possible now for many framework settings. See the table at the end of this section for more information.

For a scenario-specific configuration, XLT requires you to prefix a framework property name with the scenario name or class in the settings of your test suite. Let’s use the think time between actions as an example:

com.xceptance.xlt.thinktime.action = 5000  
TOrder_DE.com.xceptance.xlt.thinktime.action = 10000  
posters.loadtest.tests.TRestApi.com.xceptance.xlt.thinktime.action = 0

In this case, the general think time is 5,000ms, but will be 10,000ms for TOrder_DE and 0ms for TRestApi.

When looking up a property value for a scenario, XLT tries the following alternatives in this order:

1. <scenario-name>.<property-name> - the property name prefixed with the transaction name (or the simple test class name if no transaction name mapping is present)
2. <test-class-name>.<property-name> - the property name prefixed with the fully qualified test class name
3. <property-name> - the bare property name, without any prefix

See below for the list of currently supported framework properties and their default value. Properties in gray denote settings that do not support scenario-specific overrides (yet).

Load Testing

Report pages become usable much faster

Some pages in the load test report are quite lengthy, especially the Requests page. When opening such a page in the browser, it might take a while until the page is all set up and ready to use. That’s why we have optimized the JavaScript code on the page so that it takes much less time for the page to become usable than before.

AWS data center in Ohio

Recently, Amazon opened a new data center in Ohio (us-east-2 ). This new data center is now fully supported in ec2_admin, your tool to manage your AWS EC2 machine instances.

Script Developer

Script Developer has been made compatible with the latest available Firefox version, while outdated versions are not supported any longer. Script Developer runs on Firefox 45/ESR up to 50 now.

Update Instructions: Firefox will not auto-update older versions of Script Developer to 4.8.0. You will need to do this manually. Please remove the currently installed version first and afterwards install the new version by simply dragging and dropping the file <xlt>/tools/xlt-scriptdeveloper-4.8.0.xpi onto Firefox. Auto-updating within the 4.8.x product line will then work as usual again.

Alternatively, you can visit the Mozilla Add-On marketplace and install Script Developer - https://addons.mozilla.org/en-US/firefox/addon/xceptance-script-developer/. Please make sure you uninstall your locally installed version first.

XLT Jenkins Plug-In

The XLT plug-in for Jenkins has been updated as well. It comes with small improvements for users who drive their automated load tests from machines in the AWS EC2 cloud. Now the new AWS data centers in Mumbai and Ohio are fully supported, and the list of available instance types has been updated as well.

Posters Store

There is also a new version of the Posters store, our demo application shipped with XLT used to demonstrate its features. It has not only got a face-lift to look more modern, but also provides some functional improvements. All test cases in <xlt>/samples/testsuite-posters/ have been adjusted accordingly.

In case you want to update an existing installation with just the new posters.war file, you will also need to delete the database of the Posters application because the data model has slightly changed. To this end, simply delete the directory <app-server>/data/posters. The database will be recreated automatically when you start Posters the next time.

Test suite for WebDAV servers

It’s now possible to load test WebDAV servers. We have created a small library that sits on top of XLT and provides the usual WebDAV commands as predefined action classes. Simply compose your test scenarios from these configurable building blocks. The library executes the corresponding WebDAV operations and takes care of gathering all the request/response details and timings.

We also provide a demo WebDAV test suite. It does not only show how to include the library in a test suite project, but also comes with a simple scenario that demonstrates the usage of all available WebDAV actions.

The source code for both the library and the demo test suite is available on Github under the MIT License. If you just want to create test scenarios, simply clone the demo test suite. Unless you need to modify or extend the library, you won’t have to clone it too. If you need to do so nevertheless, please let us know, so all XLT users can benefit from your improvements. Pull request are welcome!

To conveniently try out the demo test suite, the demo application server that ships with XLT has been enabled to act as a WebDAV server. The WebDAV server application is available at http://localhost:8080/webdav/ and https://localhost:8443/webdav/, respectively. In the latter case, make sure to configure your WebDAV client to accept self-signed certificates. In order to get access, you need to provide valid user credentials (use webdav for both, username and password). The root directory used by the WebDAV server to serve all the files is located at <xlt>/samples/app-server/data/webdav. If this directory is not present, it will be created automatically.

Incompatible Changes

Java 8 runtime required

One of XLT’s main dependencies, Selenium, requires Java 8 now, so we have to make this step as well and ship XLT compiled for Java 8. Since Oracle discontinued the general support of Java 7 in April 2015, you are most likely using Java 8 already, so this should not be an issue for the majority of users.

Timer data

As outlined above, the timer-wd-<sessionid>.csv files with the request and browser event timings don’t exist anymore. This data can now be found in the timers.csv file. Furthermore, certain browser events recorded for Firefox have been renamed:

• DOMContentLoaded → DomContentLoadedEventEnd
• load → LoadEventEnd