Deployment Options

Last modified: April 18, 2024

1 Introduction

This document describes the supported deployment scenarios for the Application Test Suite (ATS).

Every customer situation is different. We want to make sure there’s a convenient deployment option for ATS in all scenarios. The main determinant is where your app is running, either in the cloud or on-premises. The second determinant is the Selenium component, which either can run in the cloud or be hosted on-premises by the customer. For each scenario, we need to ensure secure and reliable communication between the components.

All in all, there are four different supported deployment scenarios. This document describes and compares them.

Please be aware that not every option supports the full set of features. Take a look at Comparing the Options to understand the differences.

2 Components

Every ATS deployment consists of four components: the ATS application, the Mendix Developer Portal, a Selenium Runner, and your Application Under Test (AUT).

The following diagram illustrates how the components are connected:

2.1 Application Test Suite (ATS) Application

The ATS application provides the GUI to create, manage, and run your tests.

2.2 Mendix Developer Portal

The Mendix Developer Portal is provided as a cloud service by Mendix. It hosts your projects, user stories, and other related data that is also used by the ATS application.

2.3 Selenium Runner

The Selenium Runner is used to execute your tests. Once you trigger a test run from ATS, ATS will connect to your Selenium Service provider to start the test. The Selenium Runner then starts the browser and executes the commands that it receives from ATS.

2.4 Application Under Test (AUT)

The AUT is the deployed Mendix application that you want to test.

3 Deployment Options

There are three deployment options. The first option with the SaaS Selenium provider is recommended.

3.1 SaaS Selenium Provider

This option is highly recommended for all customers. It supports all current and future features, is easy to set up, annd is maintenance-free (the SaaS provider takes care of that). It also gets the best support.

3.2 Public App On-Premises

This option is meant for customers who run their app on premises. Since the app runs on premises, it is protected from external access via a firewall. In order to allow the Selenium Runner to access the app, you have to configure your firewall.

3.3 Private App On-Premises

This option is meant for customers who run their app on premises and who don’t want to open their firewall for the Selenium Runner. In order to enable communication between the Selenium Runner and your local application, you have to deploy a Selenium Saas Agent in your local network. This agent will establish a VPN tunnel to your Selenium Service Provider and route all traffic between the Selenium Runner and your application.

3.4 Customer-Hosted Selenium

This option is meant for customers who run their app on premises and who also want to run their own Selenium Runner. The challenging part of this option is that the customer needs to set up, maintain, and operate its own Selenium Runner. Because of the large effort this requires, this option should only be a last resort if no other option fits your use case.

Selenium is an open-source third-party component and there is no active support from Mendix.

Since the Selenium Runner offers fewer features compared to Selenium SaaS Providers, it is not possible to make use of the full ATS feature set.

The feature set in this option is limited.

4 Comparing the Options

Before you pick your option, you need to be aware of the differences. The following two sections will help you to make your choice.

4.1 Features

Feature	Standard	SaaS Selenium Provider	Public App On-Premises	Private App On-Premises	Customer-Hosted Selenium
Test Automation	Yes	Yes	Yes	Yes	Yes
Test Recording	Yes	Yes	Yes	Yes	Yes
OS Selection	No	Yes	Yes	Yes	No
Responsive Testing	Yes	Yes	Yes	Yes	No
Multiple Browsers¹	Limited	All	All	All	Limited³
Mobile Testing²	No	Yes	Yes	Yes	No
Mendix Platform Integration	All	All	Limited⁴	Limited⁴	Limited⁴
Live View & Video²	Yes	Yes	Yes	Yes	No
Future Proof⁵	Yes	Yes	Yes	Yes	No

¹ Only if supported by ATS.
² Feature not available yet.
³ Additional setup for every browser required.
⁴ ATS will integrate with several APIs from the Mendix Cloud to improve the testing experience. Since some of these APIs are only available for apps that run in the Mendix Cloud, certain features cannot be offered for on-premises apps.
⁵ Future features may depend on the functionality offered by the Mendix Cloud or the Selenium service providers. They may not be available if you run your app on-premises or host your own Selenium server.

4.2 Setup and Maintenance

The following matrix compares the efforts for setup and maintenance of the different options.

Aspect	Standard	SaaS Selenium Provider	Public App On-Premises	Private App On-Premises	Customer-Hosted Selenium
Firewall Setup	No	None¹	Required	None¹	Required
Selenium² Runner Setup	No³	No³	No³	No³	Customer responsibility
Selenium² SaaS Agent Setup	No	No⁴	No⁴	Yes	No⁴
Selenium² Maintenance	No	Yes³	Yes³	Yes³	Customer responsibility
Selenium² Support	No	Yes³	Yes³	Yes³	No

¹ A Selenium SaaS Agent with VPN is used to surpass the company network firewall.
² Selenium is an open-source third-party component that is not maintained/supported by Mendix.
³ Provided by your Selenium SaaS provider.
⁴ Not required in this option.

4.3 Uploading Files

ATS does not support the uploading of files in every situation. This table presents a summary of the different possibilities:

Selenium Setup	Uploading Your Own File	Uploading a File	Uploading Possible?
Local Selenium Server (Docker)	Limited¹	Yes	Yes
BrowserStack (SaaS)	No	Yes	Yes
SauceLabs (SaaS)	No	No	No
Selenium SaaS Agent	Limited²	Yes	Yes

¹ This is only possible when you prepare your own files on that server.
² This depends on where the agent is installed.

For more information, see How to Upload a File in Your App Using ATS.

5 Selenium SaaS Subscription

5.1 General Solutions

This section presents generic advice on which subscription type you need when using a Selenium SaaS provider. Always look at what fits best for your situation!

Selenium SaaS providers in general provide the following three solutions:

Live testing – With live testing, you can manually test your application using a virtual machine that resembles a chosen device. This option is not needed for ATS, since it is an automated testing tool. ATS only supports automated browser testing, so this solution is not viable.
Automated testing – With automated testing, you use the Selenium SaaS provider servers to execute your test cases. The provider hosts Selenium/Appium and enables all kind of extras. This is the solution you need when using ATS.
Mobile testing – With mobile testing, you use the Selenium SaaS provider servers to execute your mobile test cases. The providers host Appium and enable all kind of extras. ATS is only for browser testing, so this solution is not viable.

To test your application for desktops and mobile, you need automated browser testing.

ATS has two supported Selenium providers (for details, see Supported Selenium Providers). The solutions of these providers are discussed in the next sections.

5.2 Browserstack Solutions/Products

Browserstack is a third party, so the information displayed here may be out of date! This section is purely to help you understand the different solutions.

These Browserstack solutions are compatible with ATS:

Automate Pro – unlimited automated browser testing for desktop sites
- If you only want to test your application on desktop browsers, you should select this option
- For more information, see Browserstack Automate
Automate Mobile – unlimited automated browser testing for desktop and mobile sites
- If you want to test your application on desktop and mobile browsers, you should select this option
- For more information, see Browserstack Automate
Enterprise – personalized pricing

For more information on pricing, see Browserstack Pricing.

5.3 SauceLabs Solutions/Products

SauceLabs is a third party, so the information displayed here may be out of date! This section is purely to help you understand the different solutions.

These SauceLabs solutions are compatible with ATS:

Automated – limited browser testing for desktop and mobile sites
- If you want to test your application on desktop and mobile browsers, you should select this option
- For more information, see SauceLabs Automated
Unlimited Automated – unlimited browser testing for desktop and mobile sites
- If you want to test your application on unlimited desktop and mobile browsers, you should select this option
- For more information, see SauceLabs Automated
Enterprise – personalized pricing

There is also a difference between virtual machines and real devices, which does not matter for ATS.

For more information on pricing, see SauceLabs Pricing.

6 Setup Instructions

6.1 Standard

There are no additional steps required to set up the standard option.

6.2 Public App On-Premises

This option requires configuring your firewall in order to allow Selenium to establish a connection to your AUT.

6.2.1 Firewall Configuration

The firewall should accept connections from the internet either on port 80 (if you use http) or port 443 (if you use https) and forward to the web server of your Mendix application.

6.3 Private App On-Premises

This option requires you to deploy an agent component on-premise.

6.3.1 Agent Setup

The setup of the agents depends on your provider.

Selenium Service	Agent Name	Setup Instructions
BrowserStack	BrowserStack Local	Here
SauceLabs	Sauce Connect Proxy	Here

6.4 Customer-Hosted Selenium

This option requires you to set up your own Selenium Server on-premises.

Selenium is an open-source third-party component and there is no active support from Mendix.

Selenium standalone support has been dropped with ATS 2.0. Only Selenium Grid is supported. By following the instructions below, you will set up a Selenium Grid.

6.4.1 Selenium Runner Setup

We provide basic installation instructions for the Selenium Runner based on Docker containers. For details, see Get Started with Docker, Selenium Documentation, and Docker-Selenium.

The current ATS version requires Selenium version 3.141.59. If ATS upgrades to a newer Selenium version, the customer is obligated to upgrade its Selenium Runner to the new version as well.

Before you start, make sure that your server has internet access to download the container images.

Install Docker on your server, as described in Install Docker Engine.
Install Docker Compose, as described in Install Docker Compose.
Create a folder called selenium in the user’s home directory.

Change to the folder and create a file called docker-compose.yml with this content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


seleniumhub:
    image: selenium/hub:3.141.59
    ports:
    - 4444:4444

firefoxnode:
    image: selenium/node-firefox-debug:3.141.59
    ports:
    - 5900
    links:
    - seleniumhub:hub
    environment:
    - SCREEN_HEIGHT=1080
    - SCREEN_WIDTH=1920
    - SCREEN_DEPTH=24

chromenode:
    image: selenium/node-chrome-debug:3.141.59
    ports:
    - 5900
    links:
    - seleniumhub:hub
    environment:
    - SCREEN_HEIGHT=1080
    - SCREEN_WIDTH=1920
    - SCREEN_DEPTH=24

Start up the Selenium grid with this command:
1

docker-compose up -d
You now have a grid running on http://myserver:4444/wd/hub. Attached to this grid are two nodes: one for Chrome and one for Firefox. You can easiliy scale by starting new nodes with a simple command. This example will start up a second node for both Firefox and Chrome:
1

docker-compose scale firefoxnode=2 chromenode=2

6.4.2 Firewall Configuration

The firewall should accept TCP connections from the internet on port 4444 (the standard port for Selenium Server). This port should forward to your Selenium Server.

Feedback

Was this page helpful?

Glad to hear it! Thank you for your response.

Sorry to hear that. Please tell us how we can improve.