This document describes the supported deployment scenarios for the Application Test Suite (ATS).
Every customer situation is different. We want to make sure there’s a convenient deployment option for ATS in all scenarios. The main determinant is where your app is running — either in the cloud or on-premises. The second determinant is the Selenium component, which either can run in the cloud or be hosted on-premises by the customer. For each scenario, we need to ensure secure and reliable communication between the components.
All in all, there are four different supported deployment scenarios. This document describes and compares them.
Please be aware that not every option supports the full set of features. Take a look at 4 Comparing the Options to understand the differences.
Every ATS deployment consists of four components: the ATS application, the Mendix Cloud Portal, a Selenium Runner, and your Application Under Test (AUT).
The following diagram illustrates how the components are connected:
2.1 Application Test Suite (ATS) Application
The ATS application provides the GUI to create, manage, and run your tests.
2.2 Mendix Cloud Portal
The Mendix Cloud Portal is provided as a cloud service by Mendix. It hosts your projects, user stories, and other related data that is also used by the ATS application.
2.3 Selenium Runner
The Selenium Runner is used to execute your tests. Once you trigger a test run from ATS, ATS will connect to your Selenium Service provider to start the test. The Selenium Runner then starts the browser and executes the commands that it receives from ATS.
2.4 Application Under Test (AUT)
The AUT is the deployed Mendix application that you want to test.
3 Deployment Options
There is one standard deployment option in addition to three alternative options. The standard option is recommended, since it is optimized in terms of setup, ease of use, maintenance, and features.
The standard option is highly recommended for all customers. It supports all current and future features, is easy to set up, is maintenance-free, and gets the best support.
3.2 Public App On-Premises
This option is meant for customers who run their app on premises. Since the app runs on premises, it is protected from external access via a firewall. In order to allow the Selenium Runner to access the app, you have to configure your firewall.
3.3 Private App On-Premises
This option is meant for customers who run their app on premises and who don’t want to open their firewall for the Selenium Runner. In order to enable communication between the Selenium Runner and your local application, you have to deploy a Selenium Saas Agent in your local network. This agent will establish a VPN tunnel to your Selenium Service Provider and route all traffic between the Selenium Runner and your application.
3.4 Customer-Hosted Selenium
This option is meant for customers who run their app on premises and who also want to run their own Selenium Runner. The challenging part of this option is that the customer needs to set up, maintain, and operate its own Selenium Runner.
Since the Selenium Runner offers fewer features compared to Selenium SaaS Providers, it is not possible to make use of the full ATS feature set.
4 Comparing the Options
Before you pick your option, you need to be aware of the differences. The following two sections will help you to make your choice.
|Feature||Standard||Public App On-Premises||Private App On-Premises||Customer-Hosted Selenium|
|Mendix Platform Integration||All||Limited4||Limited4||Limited4|
|Live View & Video2||Yes||Yes||Yes||No|
1 Only if supported by ATS.
2 Feature not available yet.
3 Additional setup for every browser required.
4 ATS will integrate with several APIs from the Mendix Cloud to improve the testing experience. Since some of these APIs are only available for apps that run in the Mendix Cloud, certain features cannot be offered for on-premises apps.
5 Future features may depend on the functionality offered by the Mendix Cloud or the Selenium service providers. They may not be available if you run your app on-premises or host your own Selenium server.
4.2 Setup and Maintenance
The following matrix compares the efforts for setup and maintenance of the different options.
|Aspect||Standard||Public App On-Premises||Private App On-Premises||Customer-Hosted Selenium|
|Selenium2 Runner Setup||No3||No3||No3||Customer responsibility|
|Selenium2 SaaS Agent Setup||No4||No4||Yes||No4|
|Selenium2 Maintenance||Yes3||Yes3||Yes3||Customer responsibility|
1 A Selenium SaaS Agent with VPN is used to surpass the company network firewall.
2 Selenium is an open-source third-party component that is not maintained/supported by Mendix.
3 Provided by your Selenium SaaS provider.
4 Not required in this option.
5 Setup Instructions
There are no additional steps required to set up the standard option.
5.2 Public App On-Premises
This option requires configuring your firewall in order to allow Selenium to establish a connection to your AUT.
5.2.1 Firewall Configuration
The firewall should accept connections from the internet either on port 80 (if you use http) or port 443 (if you use https) and forward to the web server of your Mendix application.
5.3 Private App On-Premises
This option requires you to deploy an agent component on-premise.
5.3.1 Agent Setup
The setup of the agents depends on your provider.
|Selenium Service||Agent Name||Setup Instructions|
|SauceLabs||Sauce Connect Proxy||Here|
5.4 Customer-Hosted Selenium
This option requires you to set up your own Selenium Server on-premises.
5.4.1 Selenium Runner Setup
Before you start, make sure that your server has internet access to download the container images.
- Install Docker on your server, as described in Install Docker Engine.
- Install Docker Compose, as described in Install Docker Compose.
- Create a folder called selenium in the user’s home directory.
Change to the folder and create a file called docker-compose.yml with this content:
seleniumhub: image: selenium/hub:2.53.0 ports: - 4444:4444 firefoxnode: image: selenium/node-firefox-debug:2.53.0 ports: - 5900 links: - seleniumhub:hub environment: - SCREEN_HEIGHT=1080 - SCREEN_WIDTH=1920 - SCREEN_DEPTH=24 chromenode: image: selenium/node-chrome-debug:2.53.0 ports: - 5900 links: - seleniumhub:hub environment: - SCREEN_HEIGHT=1080 - SCREEN_WIDTH=1920 - SCREEN_DEPTH=24
Start up the Selenium grid with this command:
docker-compose up -d
You now have a grid running on
http://myserver:4444/wd/hub. Attached to this grid are two nodes: one for Chrome and one for Firefox. You can easiliy scale by starting new nodes with a simple command. This example will start up a second node for both Firefox and Chrome:
docker-compose scale firefoxnode=2 chromenode=2
5.4.2 Firewall Configuration
The firewall should accept TCP connections from the internet on port 4444 (the standard port for Selenium Server). This port should forward to your Selenium Server.