Roku Automation

 

The Rokuality App

The Rokuality App operates in 2 modes within a standalone app for Mac and Windows: First as a test debugger/builder which allows you to debug your apps and construct your tests, and Second, as a server which can be used to execute/distribute your tests to your devices.

StartServer.png
ServerListening.png

Note that you can start and run the server headlessly if you download the standalone jar and provide the serveronly=true system property during launch. Useful if running on a linux machine or if running from a build/test server environment:

java -Dserveronly=true -Dport=7777 -jar /path/to/Rokuality_version.jar

Optionally, you can provide a desired set of capabilities and start a manual test session against your device. The test debugger includes a number of UI tools that are geared to help you construct your automated tests. Additionally the tool can be used to provide remote access to your devices or share devices remotely across remote resources.

NewTest.png
Test.png
 

The Rokuality Platform

The Rokuality open source platform is a rich tool-set for you to write robust, end to end automation tests in a language of your choice! And it is the first framework to include support for the new Roku WebDriver API! Simply choose a supported language from below and write your tests. Then download the Rokuality app and launch a server instance to route your tests to your devices.

Available Languages:

Java

Python

C#

Javascript

How to get the Rokuality language bindings:

MAVEN

<dependency>

    <groupId>com.rokuality</groupId>

    <artifactId>rokuality-java</artifactId>

    <version>1.5.4</version>

    <scope>test</scope>

</dependency>

GRADLE

implementation 'com.rokuality:rokuality-java:1.5.4'

Roku Device Requirements and Setup

 

Automated testing on Roku requires that you have Developer mode enabled on your device. Enabling developer mode on your Roku device is very straight forward. Keep track of your device username and password as created during the basic walkthrough as you'll need them to pass to your DeviceCapabilities at driver startup. Once you've enabled developer mode on your device you should be able to hit the device console page at http://yourrokudeviceip - Once that's done you are all set for automation!

Note that some users may experience memory issues on their device under test in certain conditions. If you experience problems on driver start you can likely resolve by rebooting your Roku, or by setting your device resolution to 720p.

 

Starting a Driver and Connecting to your Device

Once you've added the bindings of your choice to your test project, and you've installed the Rokuality app and it is listening on available port, you can initiate a new Driver instance and connect to your device under test.

// your server ip address and listening port

String serverUrl = "http://your_server_ip_address:your_server_port_number";

 

// initiate your driver

RokuDriver driver = new RokuDriver(serverUrl, DeviceCapabilities);

 

Device Capabilities on Session Start

A capability object as passed to your driver declaration will control certain functionality on test start. For Roku devices, the only required capabilities can be seen below. Others are optional.

// your server ip and listening port

String serverUrl = "http://your_server_ip:your_listening_port";

// Declare a new DeviceCapability object

DeviceCapabilities capabilities = new DeviceCapabilities();

 

// Indicates we want a Roku test session

capabilities.addCapability("Platform", "Roku");

 

// App location (path or url to a sideloadable zip)

capabilities.addCapability("AppPackage", "/path/or/url/to/your/sideloadable/app.zip");

// Your Roku device ip address

capabilities.addCapability("DeviceIPAddress", "your_roku_ip_address");

// Your Roku developer mode username and password

capabilities.addCapability("DeviceUsername", "rokudev");

capabilities.addCapability("DevicePassword", "your_dev_setting_password");

 

// Pass the capabilities and start the test

RokuDriver rokuDriver = new RokuDriver(serverUrl, capabilities);

All Device Capabilities

 

Platform

Required - String Value. Indicates the target platform. For Roku automation, required value is Roku.

 

AppPackage

Required - String Value. The absolute path or url to a Roku sideloadable .zip package to be installed on the device.

 

DeviceIPAddress

Required - String Value. The IP address of your Roku device. Note that the Roku must be on the same network as your Rokuality server.

 

DeviceUsername

Required - String Value. The dev console username created when you enabled developer mode on your device.

 

DevicePassword

Required - String Value. The dev console password created when you enabled developer mode on your device.

 

AppLaunchParameters

Optional - String Value. A comma separated string of parameters and values in the format param=value that can be passed to the application on launch. i.e. "param1=value1,param2=value2"

 

ImageMatchSimilarity

Optional - Double Value. An optional image match similarity default used only during Image locator evaluations. A lower value will allow for greater tolerance of image disimilarities between the image locator and the screen, BUT will also increase the possibility of a false positive. Double. Defaults to .90

 

ScreenSizeOverride

Optional - String Value. An optional 'WIDTHxHEIGHT' cap that all screen image captures will be resized to prior to match evaluation. Useful if you want to enforce test consistency across multiple device types and multiple developer machines or ci environments. String - i.e. a value of '1800x1200' will ensure that all image captures are resized to those specs before the locator evaluation happens no matter what the actual device screen size is.

 

OCRType

Optional - String Value. The OCR type - Currently supported options are 'Tesseract', 'GoogleVision', 'AmazonRekognition', or 'AmazonTextract'. If the capability is set to 'GoogleVision' you MUST have a valid Google Vision account setup and provide the 'GoogleCredentials' capability with a valid file path to the oath2 .json file with valid credentials for the Google Vision service. If the capability is set to 'AmazonRekognition' or 'AmazonTextract' then you MUST have a valid AWS account with an IAM role set with Rekognition or Textract priveleges and an AWS api access key id and secret key in file format you can provide, and you MUST provide the 'AWSCredentials' capability with a valid file path to this credentials file. If the capability is set to 'Tesseract', then you MUST have tesseract installed on your machine. See the using OCR section for details.

 

GoogleCredentials

Optional - String Value. The path to a valid .json Google Auth key service file. Required if the 'OCRType' capability is set to 'GoogleVision'. The .json service key must exist on the machine triggering the tests, and the Google account for the service must have permissions for Cloud Vision API. The .json service key must exist on the machine triggering the tests. See the using OCR section for details.

 

AWSCredentials

Optional - String Value. The path to a valid AWS credential file with an api key id and secret key. Optional but Required if the 'OCRType' capability is set to 'AmazonRekognition' or 'AmazonTextract'. The credential file must exist on the machine triggering the tests. See the using OCR section for details.

 

EnablePerformanceProfiling

Optional - Boolean Value. If provided your sideloadable .zip package will be updated for performance profiling. Then during the course of the execution you can retrieve the .bsprof file with CPU and memory utilization data from your channel. Boolean - true to allow for performance capturing. Defaults to false.

 

ContentID

Optional - String Value. A content id of a Roku deep link to load on session start. If provided you must also provide the 'MediaType' capability.

 

MediaType

Optional - String Value. The media type of a Roku deep link to load on session start. If provided you must also provide the 'ContentID' capability.

 

ImageCollectorInterval

Optional - Integer Value. A value in milliseconds that acts as a delay between image collection during a test session. If you experience 400 http errors during image collection, a small pause here can help alleviate this. A delay in milliseconds i.e. 250. Defaults to 0

 

ProxyHost

Optional - String Value. The IP address of the machine running the Rokuality server. If this capability is passed in combination with the ProxyPort capability, then a Proxy will be started and the underlying Roku application traffic will route through it - allowing your to monitor and modify your Roku http(s) traffic during test execution. See the proxy section for details.

 

ProxyPort

Optional - Integer Value. The port number you wish to start a proxy on. Must be an open port for every proxy instance you wish to start. If this capability is passed in combination with the ProxyHost capability, then a Proxy will be started and the underlying Roku application traffic will route through it - allowing your to monitor and modify your Roku http(s) traffic during test execution. See the proxy section for details.

 

ProxyType

Optional - String Value. The type of Proxy you wish to use. We currently support Charles Proxy - Windows and Mac, and Fiddler Classic - Windows. See the proxy section for details. Valid values are Charles or Fiddler. If this capability is ommitted but the user provides the ProxyPort and ProxyHost capabilities then we will attempt to determine which proxy is installed and use the appropriate proxy. If you have both Fiddler and Charles installed on your machine you must provide this capability so Rokuality knows which proxy to use.

 

ProxyBinary

Optional - String Value. The absolute path to your Charles or Fiddler binary. See the proxy section for details. If this capability is omitted but the user provides the ProxyPort and ProxyHost capabilities then we will attempt to find the Charles or Fiddler binary executable in the default installation locations.

 

HeadlessProxy

Optional - Boolean Value. If set to false, then the proxy gui will be visible to the user. See the proxy section for details. Defaults to false - i.e. the proxy gui is not visible during execution and runs in the background.

 

TesseractBinary

Optional - String Value. If using the 'OCRType' capability with value 'Tesseract', this capability must be provided with the absolute path to your Tesseract binary as installed on you machine, i.e. '/usr/local/bin/tesseract'. See the using OCR section for details.

 

TesseractLanguage

Optional - String Value. If using the 'OCRType' capability with value 'Tesseract', this capability can be provided with the 3-character ISO 639-2 language code you wish to use, i.e. 'eng'. See the using OCR section for details. Defaults to 'eng'.

 

Finding Elements During Test

During your test, you can identify and locate elements in a variety of ways:

Roku WebDriver Locators

The Rokuality platform is one of the first to provide support for the new Roku WebDriver API! You can access native based locators (Text, Tag, Attribute) during the course of your test as below. Additionally, Rokuality extends the webdriver API and provides support for XPath location as well.

// Finds an element by the native based text of the element

Element eleByTxt = driver.finder().findElement(RokuBy.Text("text"));

 

// Finds an element by the native tag

Element eleByTag = driver.finder().findElement(RokuBy.Tag("tag"));

 

// Finds an element by the native attribute name and value

Element eleByAtr = driver.finder().findElement(RokuBy.Attribute("name", "value"));

// Finds an element by XPath

Element eleByXPath = driver.finder().findElement(RokuBy.XPath("//xpathquery"));

OCR Text Locators

OCR - Text based locators work by capturing the screen image of your device and then performing an evaluation to determine if the text resides within the image. This can be a bit less reliable than the above native locators but can prove incredibly useful for testing text within images or for testing scenarios that aren't available in the rich document object model of the Roku channel. Note that for this locator type to be available,  you must provide the OCRType capability with all the necessary requirements. See the device capabilities section and the using OCR section for additional details.

Element element = driver.finder().findElement(By.Text("text to find on screen"));

Image Snippet Locators

Image snippet locators allow you to provide a partial image snippet you would expect to exist within the device screen. The device screen is then captured and searched to see if it contains the expected image snippet. This is very useful if you wish to verify images/colors/logos/etc in your application. But be cautioned as this is the most fragile of all locator types as the image snippet capture must reliably match the screen of the device for the evaluation. See the Device Capabilities section above as the 'ImageMatchSimilarity' and 'ScreenSizeOverride' capabilities can help with this. But you still need to ensure that the image snippet you're passing is an apples to apples comparison against the device screen.

Note that your image snippet MUST be in .png format and can be either the absolute path to an image snippet on your machine, or can be the url to an image snippet that it can access. The latter is useful if you wish to query your application image content from a remote content api and then dynamically search for them within your test.

// Finds an element by a .png image snippet saved on the users file system

Element eleFromFile = driver.finder().findElement(By.Image("/path/to/image.png"));

// Finds an element by a .png image snippet available at a public url

Element eleFromUrl = driver.finder().findElement(By.Image("http://urltoimage.png"));

Element Timeouts and NoSuchElement Exceptions

In the event our locator can't be found within the application, a NoSuchElementException will be returned to the user and the test will fail. In the above scenarios this failure would happen immediately as we did not apply an implicit wait for our locator searches. But if we apply a timeout (in milliseconds), we can reduce flake in our tests as the locator will be searched for continuously until it is either found, or the timeout expires and a NoSuchElementException is thrown. The implicit element timeout will last for the duration of the driver session, or until a new value is set that overrides it.

// Sets an element timeout applied to all locator searches

// if set the server will look for the element until it is either found

// or the timeout is exceeded and a NoSuchElementException is thrown

driver.options().setElementTimeout(5000);

driver.finder().findElement(RokuBy.Text("text to find"));

 

Elements as Objects

Once our locator has been found, a matching Element object will be returned which will contain information about its location, size, and confidence (if relevant).

Element element = driver.finder().findElement(By.Text("Hello World!"));

System.out.println(element.getLocation());

System.out.println(element.getHeight());

System.out.println(element.getWidth());

System.out.println(element.getConfidence());

System.out.println(element.getText());

 

Finding Multiple Elements and Checking for Element Presence

In the above scenarios, we examine searching for a single element within our application. But in those cases, if the element is not found by the designated locator, then a NoSuchElementException will be thrown and our test will fail. But what if we want to check if an element is present or not? Or our locator in question returns multiple elements on the device screen?

Multiple Match Locators

Locators that find multiple matches for an element will return a collection of those elements which can be iterated over as follows:

List<Element> elements = driver.finder().findElements(By.Text("multi match locator"));

Element Presence

This same approach can be used to check whether an element is present within the application. Using the multi match search will NOT throw a NoSuchElementException in the event a matching element is not found. In that event the collection will be empty and we can perform logic based on that scenario.

boolean elementPresent = driver.finder().findElements(By.Text("locator")).size() > 0;

Remote Control and User Interaction

 

A user can perform remote control button presses and drive the application as a user would with the remote control api's. A user can navigate the UI of their app, and pause/play/fast forward/rewind media in flight.

driver.remote().pressButton(RokuButton.SELECT);

Almost all remote control buttons are supported with the exception of the home remote button as users are only allowed to perform remote control interactions while their application under test is in focus.

Java

Python

C#

Javascript

In addition to remote control button presses, it is also possible to send a string of literal characters to the device to perform searches or easily interact with the Roku keyboard. Note that the Roku virtual keyboard must be visible on screen for this to have any effect.

driver.remote().sendKeys("text to type");

Getting Screen Artifacts (Image, Recordings, Page Source, and More)

 

During the course of a test, a user can get screen artifacts such as the screen image, screen sub image, and xml page source. It's also possible to get the screen recording of the device from session start until the time of capture which is incredibly useful for reporting and test debugging.

// get the screen size

driver.screen().getSize()

 

// get the screen image

driver.screen().getImage();

 

// get the screen sub image from starting x,y with width/height

driver.screen().getImage(1, 1, 300, 300);

 

// get the screen recording of the test session from start to now

driver.screen().getRecording();

 

// gets the xml page source of the Roku channel.

// useful for constructing Roku native text/tag/attribute locators

driver.screen().getPageSource();

 

// gets the currently focused roku element

driver.screen().getActiveElement();

Note that while the Roku media player is focused on the screen, the screen canvas will be blacked out as a content protection. This is enforced by the underlying Roku developer api's used during image capture. As a result, while the media player is on screen, OCR and Image based locators will not be available. But the native based Roku WebDriver locators will still be applicable. Also, additional details about the media player can be retrieved from the Media Player API.

Getting Information About the Device

 

Information about the device under test such as the device type, model number, and more can be retrieved during the session as follows:

RokuDeviceInfo deviceInfo = driver.info().getDeviceInfo();

Getting Media Player Information from the Device

 

Information from the media player such as playback state, buffering rate, errors, and much much more can be retrieved during the test session. The returned media player information is invaluable for verifying the health of your media in flight.

// gets information about the media in flight including state, bitrate, encoding, etc.

RokuMediaPlayerInfo mediaPlayerInfo = driver.info().getMediaPlayerInfo();

 

Assert.assertFalse(mediaPlayerInfo.isError());

Assert.assertFalse(mediaPlayerInfo.isLive());

Assert.assertEquals(mediaPlayerInfo.getState(), "play");

Setting and Retrieving Session Status

 

You can set the status of an active session to "passed", "failed", "broken", or "in progress" which will be retained in server memory for the duration of the session or until a new value is set. This is useful if you want to set the status of a test and then communicate result status with a reporting framework/service in a teardown or after test method. By default, the session status is "in progress" unless the user has updated it during the course of the session.

// sets the session status

driver.options().setSessionStatus(SessionStatus.PASSED);

// can be retrieved at any point the session is active

SessionStatus status = driver.options().getSessionStatus();

Assert.assertEquals(SessionStatus.PASSED, status);

Properly Stopping Your Session on Test Complete

 

It's important that when your test is complete, you properly stop your driver and release your device! This terminates the session under test and frees up the available thread for additional testing. If you don't properly release the device, back end cleanup will eventually run and release the device for further testing.

// stops the driver and releases your available thread back to your plan

// should be called as the last action of your test

driver.stop();

Collecting CPU and Memory Performance Data During Test

 

It is possible to access the Roku brightscript profiler data for CPU and Memory analysis during the course of your test execution. Prior to session start, provide the 'EnablePerformanceProfiling' capability with a value of true. This will ensure your provided sideloadable .zip package is recompiled for performance monitoring prior to application launch. Then, at any point during the course of your test execution, you can collect the .bsprof profile file containing valuable details about CPU and Memory utilization. This file can be loaded into Roku's Brightscript Profiler Visualization Tool for a detailed breakdown of your app's performance from test start to collection.

// set your 'EnablePerformanceProfiling' capability to true and initiate your driver

capabilities.addCapability("EnablePerformanceProfiling", true);

driver = new RokuDriver(SERVER_URL, caps);

 

// now at any point during the test we can collect the performance profile

File perfProfile = driver.info().getPerformanceProfile();

Testing Deep Links

 

Deep link testing is possible by providing the 'ContentID' and 'MediaType' capabilities on session start. If provided, the channel will be launched with the provided deep link location loaded. If an error occurs and the deep link cannot be loaded, then a 'SessionNotStartedException' will be thrown.

capabilities.addCapability("ContentID", "idofcontent");

capabilities.addCapability("MediaType", "mediatypeofcontent");

driver = new RokuDriver(SERVER_URL, caps);

Roku Proxy with SSL Support

 

Rokuality is the first platform to provide proxy support with ssl decryption for Roku devices! No need to tether your Roku to your machine via wifi sharing and no need to perform complicated router based modifications! See the following sections for details on proxy setup and usage.

Getting Started

To get started monitoring/modifying your Roku http(s) traffic during test, you must install one of our 2 supported proxies. Rokuality current supports Charles Proxy - Windows and Mac and Fiddler Classic. Both require some simple 1 time setup after you have installed the proxy of your choice.

Charles Proxy - First Time Setup

  1. Install Charles Proxy version 4.5.6 or newer.

  2. Launch Charles Proxy for the first time.

  3. Optional - If you wish to run multiple instances of Charles Proxy while testing, i.e. test concurrency with each test having their own Charles Proxy instance, please see these instructions

This should be all that is required for Rokuality to be able to create Charles Proxy configurations and launch/stop/query the proxy as needed during automation.

Fiddler Classic - First Time Setup

  1. Install the latest version of Fiddler Classic.

  2. Launch Fiddler Classic for the first time.

  3. Create a CustomRules.js file for the first time use by choosing the toolbar menu Rules>>Customize Rules.... This will create the backend CustomRules.js file that Rokuality needs in order to launch/stop/query the proxy during automation.

  4. Allow remote connections from your device to the proxy by choosing the toolbar menu Tools>>Options>>Connections and check the Allow remote computers to connect checkbox.

  5. Disable the system proxy on startup by choosing the toolbar menu Tools>>Options>>Connections and un-check the Act as a system proxy on startup checkbox.

  6. Optional - if you wish to decrypt ssl traffic during test - Enable https connections by choosing the toolbar menu Tools>>Options>>HTTPS and check both the Capture HTTPS CONNECTs checkbox and the Decrypt HTTPS traffic from all processes checkbox. You may also wish to ensure that all protocols are entered for capture including <client>;ssl2;ssl3;tls1.0;tls1.1;tls1.2

Note that when using Fiddler - you may only have 1 instance of Fiddler running at one time, i.e. you may only run 1 test per machine at any one time. You can scale this by distributing your tests to multiple Rokuality servers but if you need concurrency, Charles Proxy might be a better solution for you.

Starting a Proxy

To start a proxy during test execution, you must provide the minimum required capabilities which include the ProxyHost and ProxyPort capabilities. See the capabilities section for additional capabilities. When the minimum caps for a Proxy connection are provided, a Charles or Fiddler proxy will be started on the backend and all traffic from your Roku device will then route through the proxy.

// Declare a new DeviceCapability object

DeviceCapabilities capabilities = new DeviceCapabilities();

 

// The ip address of your machine running Rokuality which allows your

// Roku to communicate with your Rokuality

// server and proxy server, i.e. 192.168.1.47

capabilities.addCapability("ProxyHost", "your server ip address");

 

// Any open port on your machine you wish the proxy to run on.

// Each test session must have a unique port

capabilities.addCapability("ProxyPort", 8899);

 

// Pass the capabilities and start the test

RokuDriver driver = new RokuDriver("http://ipaddressofyourrunningserver:port", capabilities);

Roku Proxy: Monitoring http(s) Calls During Test

During the course of a test, the proxy api will allow you to retrieve your request/response traffic in standard har format. Several libraries exist that will allow you to parse your har content into objects during test, but at the core a har file is a json file that can be cast/queried as any JSONObject.

// gets the har file of every request in the proxy from test start to now

File harFile = driver.proxy().getHarLog();

System.out.println("Har file saved to: " + harFile);

 

// clear the har log

driver.proxy().clearHarLog();