Multi-device test automation at ticketea

Nowadays, software Quality Assurance is all about automatic processes and continuous integration. At ticketea, we chose some of the leading and more standard tools to build our QA toolchain. We work with Selenium and Chrome Webdriver, along with Python 3.6. The advantage of this stack is that it is popular, growing and maintained by a large community.

For starters, we wanted to be sure that our most important and vital processes (such as the payment and event creation funnels) were QA'd automatically whenever new code was pushed.

Smoke Tests and Full Tests

We execute a preliminary Smoke Test using a headless Chrome instance in a Docker container. This way, we have a fast feedback loop for mistakes or bugs that might have slipped into the code and have not been captured by unit tests.
We consider this a smoke test, because it's run against a single browser version. Thanks to SauceLabs, we're also able run our tests on a grid of multiple browsers and devices, as shown below:

It was easy to compile a list of common devices to be supported analyzing our traffic in Google Analytics.

Notice a couple of things: SauceLabs shows you the VPN tunnel connection indicator (the thunder icon in the 2nd column), and which browser and OS are being used for executing each test. The interesting part is that you'll need a VPN to connect Saucelabs with your target website, in case it's not publicly available.

How to run Selenium tests in Saucelabs

The following piece of code shows how we are connecting with Saucelabs to run our test suite.

from threading import local

import os  
from pyvirtualdisplay import Display  
from selenium import webdriver  
from selenium.webdriver.remote.remote_connection import RemoteConnection

_cache = local()

USE_VIRTUAL_DISPLAY = os.environ.get("USE_VIRTUAL_DISPLAY", "false") == "true"

def init_driver(mode, sl_credentials, capabilities):  
    if mode == 'appium':
        driver = webdriver.Remote(
            command_executor='http://localhost:4723/wd/hub',
            desired_capabilities={
                    "platformName":"Android",
                    "browserName":"Chrome",
                    "javascriptEnabled":True,
                    "version":"",
                    "deviceName":"MyAndroid"
            }
        )
    elif mode == 'local':
        options = webdriver.ChromeOptions()
        options.add_argument("--test-type")
        options.add_argument("--allow-running-insecure-content")
        options.add_argument("--ignore-certificate-errors")
        options.add_argument("--no-sandbox")

        if USE_VIRTUAL_DISPLAY and not hasattr(_cache, "display"):
            _cache.display = Display(visible=0, size=(1024, 768), use_xauth=True)
            _cache.display.start()
        driver = webdriver.Chrome(chrome_options=options)
    elif mode == 'saucelabs':
        url = 'http://{user}:{secret}@{url}'.format(**sl_credentials)
        driver = webdriver.Remote(
            command_executor=RemoteConnection(url, resolve_ip=False),
            desired_capabilities=capabilities)
    else:
        raise RuntimeError('Bad mode specified: {}'.format(mode))

    return driver

As you can see, depending on the specific testing mode (appium, local or saucelabs), the webdriver configuration is slightly different. By the way, there will be more on Appium (a framework aimed at testing mobile apps) in a separate post.

Conclusion

We think that the combination of Selenium, Appium, Webdriver, and SauceLabs suits our needs quite well. Thanks to this stack, we can now concentrate on our business needs and expand our automatic QA coverage.