The Rokuality App
The Rokuality App operates in 2 modes within a standalone app for Mac and Windows: First as a test debugger/builder which allows you to debug your apps and construct your tests, and Second, as a server which can be used to execute/distribute your tests to your devices.
Note that you can start and run the server headlessly if you download the standalone jar and provide the serveronly=true system property during launch. Useful if running on a linux machine or if running from a build/test server environment:
java -Dserveronly=true -Dport=7777 -jar /path/to/Rokuality_version.jar
Optionally, you can provide a desired set of capabilities and start a manual test session against your device. The test debugger includes a number of UI tools that are geared to help you construct your automated tests. Additionally the tool can be used to provide remote access to your devices or share devices remotely across remote resources.
The Rokuality Platform
The Rokuality open source platform is a rich tool-set for you to write robust, end to end automation tests in a language of your choice! And it is the first framework to include support for the new Roku WebDriver API! Simply choose a supported language from below and write your tests. Then download the Rokuality app and launch a server instance to route your tests to your devices.
How to get the Rokuality language bindings:
Roku Device Requirements and Setup
Automated testing on Roku requires that you have Developer mode enabled on your device. Enabling developer mode on your Roku device is very straight forward. Keep track of your device username and password as created during the basic walkthrough as you'll need them to pass to your DeviceCapabilities at driver startup. Once you've enabled developer mode on your device you should be able to hit the device console page at - Once that's done you are all set for automation!
Note that some users may experience memory issues on their device under test in certain conditions. If you experience problems on driver start you can likely resolve by rebooting your Roku, or by setting your device resolution to 720p.
Starting a Driver and Connecting to your Device
Once you've added the bindings of your choice to your test project, and you've installed the Rokuality app and it is listening on available port, you can initiate a new Driver instance and connect to your device under test.
// your server ip address and listening port
String serverUrl = ":your_server_port_number";
// initiate your driver
RokuDriver driver = new RokuDriver(serverUrl, DeviceCapabilities);
Device Capabilities on Session Start
A capability object as passed to your driver declaration will control certain functionality on test start. For Roku devices, the only required capabilities can be seen below. Others are optional.
// your server ip and listening port
String serverUrl = ":your_listening_port";
// Declare a new DeviceCapability object
DeviceCapabilities capabilities = new DeviceCapabilities();
// Indicates we want a Roku test session
// App location (path or url to a sideloadable zip)
// Your Roku device ip address
// Your Roku developer mode username and password
// Pass the capabilities and start the test
RokuDriver rokuDriver = new RokuDriver(serverUrl, capabilities);
All Device Capabilities
Required - String Value. Indicates the target platform. For Roku automation, required value is Roku.
Required - String Value. The absolute path or url to a Roku sideloadable .zip package to be installed on the device.
Required - String Value. The IP address of your Roku device. Note that the Roku must be on the same network as your Rokuality server.
Required - String Value. The dev console username created when you enabled developer mode on your device.
Required - String Value. The dev console password created when you enabled developer mode on your device.
Optional - String Value. A comma separated string of parameters and values in the format param=value that can be passed to the application on launch. i.e. "param1=value1,param2=value2"
Optional - Double Value. An optional image match similarity default used only during Image locator evaluations. A lower value will allow for greater tolerance of image disimilarities between the image locator and the screen, BUT will also increase the possibility of a false positive. Double. Defaults to .90
Optional - String Value. An optional 'WIDTHxHEIGHT' cap that all screen image captures will be resized to prior to match evaluation. Useful if you want to enforce test consistency across multiple device types and multiple developer machines or ci environments. String - i.e. a value of '1800x1200' will ensure that all image captures are resized to those specs before the locator evaluation happens no matter what the actual device screen size is.
Optional - String Value. The OCR type - Currently supported options are 'Tesseract', 'GoogleVision', 'AmazonRekognition', or 'AmazonTextract'. If the capability is set to 'GoogleVision' you MUST have a valid Google Vision account setup and provide the 'GoogleCredentials' capability with a valid file path to the oath2 .json file with valid credentials for the Google Vision service. If the capability is set to 'AmazonRekognition' or 'AmazonTextract' then you MUST have a valid AWS account with an IAM role set with Rekognition or Textract priveleges and an AWS api access key id and secret key in file format you can provide, and you MUST provide the 'AWSCredentials' capability with a valid file path to this credentials file. If the capability is set to 'Tesseract', then you MUST have tesseract installed on your machine. See the using OCR section for details.
Optional - String Value. The path to a valid .json Google Auth key service file. Required if the 'OCRType' capability is set to 'GoogleVision'. The .json service key must exist on the machine triggering the tests, and the Google account for the service must have permissions for Cloud Vision API. The .json service key must exist on the machine triggering the tests. See the using OCR section for details.
Optional - String Value. The path to a valid AWS credential file with an api key id and secret key. Optional but Required if the 'OCRType' capability is set to 'AmazonRekognition' or 'AmazonTextract'. The credential file must exist on the machine triggering the tests. See the using OCR section for details.
Optional - Boolean Value. If provided your sideloadable .zip package will be updated for performance profiling. Then during the course of the execution you can retrieve the .bsprof file with CPU and memory utilization data from your channel. Boolean - true to allow for performance capturing. Defaults to false.
Optional - String Value. A content id of a Roku deep link to load on session start. If provided you must also provide the 'MediaType' capability.
Optional - String Value. The media type of a Roku deep link to load on session start. If provided you must also provide the 'ContentID' capability.
Optional - Integer Value. A value in milliseconds that acts as a delay between image collection during a test session. If you experience 400 http errors during image collection, a small pause here can help alleviate this. A delay in milliseconds i.e. 250. Defaults to 0
Optional - String Value. The IP address of the machine running the Rokuality server. If this capability is passed in combination with the ProxyPort capability, then a Proxy will be started and the underlying Roku application traffic will route through it - allowing your to monitor and modify your Roku http(s) traffic during test execution. See the proxy section for details.
Optional - Integer Value. The port number you wish to start a proxy on. Must be an open port for every proxy instance you wish to start. If this capability is passed in combination with the ProxyHost capability, then a Proxy will be started and the underlying Roku application traffic will route through it - allowing your to monitor and modify your Roku http(s) traffic during test execution. See the proxy section for details.
Optional - String Value. The type of Proxy you wish to use. We currently support Charles Proxy - Windows and Mac, and Fiddler Classic - Windows. See the proxy section for details. Valid values are Charles or Fiddler. If this capability is ommitted but the user provides the ProxyPort and ProxyHost capabilities then we will attempt to determine which proxy is installed and use the appropriate proxy. If you have both Fiddler and Charles installed on your machine you must provide this capability so Rokuality knows which proxy to use.
Optional - String Value. The absolute path to your Charles or Fiddler binary. See the proxy section for details. If this capability is omitted but the user provides the ProxyPort and ProxyHost capabilities then we will attempt to find the Charles or Fiddler binary executable in the default installation locations.
Optional - Boolean Value. If set to false, then the proxy gui will be visible to the user. See the proxy section for details. Defaults to false - i.e. the proxy gui is not visible during execution and runs in the background.
Optional - String Value. If using the 'OCRType' capability with value 'Tesseract', this capability must be provided with the absolute path to your Tesseract binary as installed on you machine, i.e. '/usr/local/bin/tesseract'. See the using OCR section for details.
Optional - String Value. If using the 'OCRType' capability with value 'Tesseract', this capability can be provided with the 3-character ISO 639-2 language code you wish to use, i.e. 'eng'. See the using OCR section for details. Defaults to 'eng'.
Finding Elements During Test
During your test, you can identify and locate elements in a variety of ways:
Roku WebDriver Locators
The Rokuality platform is one of the first to provide support for the new Roku WebDriver API! You can access native based locators (Text, Tag, Attribute) during the course of your test as below. Additionally, Rokuality extends the webdriver API and provides support for XPath location as well.
// Finds an element by the native based text of the element
Element eleByTxt = driver.finder().findElement(RokuBy.Text("text"));
// Finds an element by the native tag
Element eleByTag = driver.finder().findElement(RokuBy.Tag("tag"));
// Finds an element by the native attribute name and value
Element eleByAtr = driver.finder().findElement(RokuBy.Attribute("name", "value"));
// Finds an element by XPath
Element eleByXPath = driver.finder().findElement(RokuBy.XPath("//xpathquery"));
OCR Text Locators
OCR - Text based locators work by capturing the screen image of your device and then performing an evaluation to determine if the text resides within the image. This can be a bit less reliable than the above native locators but can prove incredibly useful for testing text within images or for testing scenarios that aren't available in the rich document object model of the Roku channel. Note that for this locator type to be available, you must provide the OCRType capability with all the necessary requirements. See the device capabilities section and the using OCR section for additional details.
Element element = driver.finder().findElement(By.Text("text to find on screen"));
Image Snippet Locators
Image snippet locators allow you to provide a partial image snippet you would expect to exist within the device screen. The device screen is then captured and searched to see if it contains the expected image snippet. This is very useful if you wish to verify images/colors/logos/etc in your application. But be cautioned as this is the most fragile of all locator types as the image snippet capture must reliably match the screen of the device for the evaluation. See the Device Capabilities section above as the 'ImageMatchSimilarity' and 'ScreenSizeOverride' capabilities can help with this. But you still need to ensure that the image snippet you're passing is an apples to apples comparison against the device screen.
Note that your image snippet MUST be in .png format and can be either the absolute path to an image snippet on your machine, or can be the url to an image snippet that it can access. The latter is useful if you wish to query your application image content from a remote content api and then dynamically search for them within your test.
// Finds an element by a .png image snippet saved on the users file system
Element eleFromFile = driver.finder().findElement(By.Image("/path/to/image.png"));
// Finds an element by a .png image snippet available at a public url
Element eleFromUrl = driver.finder().findElement(By.Image(""));
Element Timeouts and NoSuchElement Exceptions
In the event our locator can't be found within the application, a NoSuchElementException will be returned to the user and the test will fail. In the above scenarios this failure would happen immediately as we did not apply an implicit wait for our locator searches. But if we apply a timeout (in milliseconds), we can reduce flake in our tests as the locator will be searched for continuously until it is either found, or the timeout expires and a NoSuchElementException is thrown. The implicit element timeout will last for the duration of the driver session, or until a new value is set that overrides it.
// Sets an element timeout applied to all locator searches
// if set the server will look for the element until it is either found
// or the timeout is exceeded and a NoSuchElementException is thrown
driver.finder().findElement(RokuBy.Text("text to find"));
Elements as Objects
Once our locator has been found, a matching Element object will be returned which will contain information about its location, size, and confidence (if relevant).
Element element = driver.finder().findElement(By.Text("Hello World!"));
Finding Multiple Elements and Checking for Element Presence
In the above scenarios, we examine searching for a single element within our application. But in those cases, if the element is not found by the designated locator, then a NoSuchElementException will be thrown and our test will fail. But what if we want to check if an element is present or not? Or our locator in question returns multiple elements on the device screen?
Multiple Match Locators
Locators that find multiple matches for an element will return a collection of those elements which can be iterated over as follows:
List<Element> elements = driver.finder().findElements(By.Text("multi match locator"));
This same approach can be used to check whether an element is present within the application. Using the multi match search will NOT throw a NoSuchElementException in the event a matching element is not found. In that event the collection will be empty and we can perform logic based on that scenario.
boolean elementPresent = driver.finder().findElements(By.Text("locator")).size() > 0;
Remote Control and User Interaction
A user can perform remote control button presses and drive the application as a user would with the remote control api's. A user can navigate the UI of their app, and pause/play/fast forward/rewind media in flight.
Almost all remote control buttons are supported with the exception of the home remote button as users are only allowed to perform remote control interactions while their application under test is in focus.
In addition to remote control button presses, it is also possible to send a string of literal characters to the device to perform searches or easily interact with the Roku keyboard. Note that the Roku virtual keyboard must be visible on screen for this to have any effect.
driver.remote().sendKeys("text to type");
Getting Screen Artifacts (Image, Recordings, Page Source, and More)
During the course of a test, a user can get screen artifacts such as the screen image, screen sub image, and xml page source. It's also possible to get the screen recording of the device from session start until the time of capture which is incredibly useful for reporting and test debugging.
// get the screen size
// get the screen image
// get the screen sub image from starting x,y with width/height
driver.screen().getImage(1, 1, 300, 300);
// get the screen recording of the test session from start to now
// gets the xml page source of the Roku channel.
// useful for constructing Roku native text/tag/attribute locators
// gets the currently focused roku element
Note that while the Roku media player is focused on the screen, the screen canvas will be blacked out as a content protection. This is enforced by the underlying Roku developer api's used during image capture. As a result, while the media player is on screen, OCR and Image based locators will not be available. But the native based Roku WebDriver locators will still be applicable. Also, additional details about the media player can be retrieved from the Media Player API.
Getting Information About the Device
Information about the device under test such as the device type, model number, and more can be retrieved during the session as follows:
RokuDeviceInfo deviceInfo = driver.info().getDeviceInfo();
Getting Media Player Information from the Device
Information from the media player such as playback state, buffering rate, errors, and much much more can be retrieved during the test session. The returned media player information is invaluable for verifying the health of your media in flight.
// gets information about the media in flight including state, bitrate, encoding, etc.
RokuMediaPlayerInfo mediaPlayerInfo = driver.info().getMediaPlayerInfo();
Setting and Retrieving Session Status
You can set the status of an active session to "passed", "failed", "broken", or "in progress" which will be retained in server memory for the duration of the session or until a new value is set. This is useful if you want to set the status of a test and then communicate result status with a reporting framework/service in a teardown or after test method. By default, the session status is "in progress" unless the user has updated it during the course of the session.
// sets the session status
// can be retrieved at any point the session is active
SessionStatus status = driver.options().getSessionStatus();
Properly Stopping Your Session on Test Complete
It's important that when your test is complete, you properly stop your driver and release your device! This terminates the session under test and frees up the available thread for additional testing. If you don't properly release the device, back end cleanup will eventually run and release the device for further testing.
// stops the driver and releases your available thread back to your plan
// should be called as the last action of your test
Collecting CPU and Memory Performance Data During Test
It is possible to access the Roku brightscript profiler data for CPU and Memory analysis during the course of your test execution. Prior to session start, provide the 'EnablePerformanceProfiling' capability with a value of true. This will ensure your provided sideloadable .zip package is recompiled for performance monitoring prior to application launch. Then, at any point during the course of your test execution, you can collect the .bsprof profile file containing valuable details about CPU and Memory utilization. This file can be loaded into Roku's Brightscript Profiler Visualization Tool for a detailed breakdown of your app's performance from test start to collection.
// set your 'EnablePerformanceProfiling' capability to true and initiate your driver
driver = new RokuDriver(SERVER_URL, caps);
// now at any point during the test we can collect the performance profile
File perfProfile = driver.info().getPerformanceProfile();
Testing Deep Links
Deep link testing is possible by providing the 'ContentID' and 'MediaType' capabilities on session start. If provided, the channel will be launched with the provided deep link location loaded. If an error occurs and the deep link cannot be loaded, then a 'SessionNotStartedException' will be thrown.
driver = new RokuDriver(SERVER_URL, caps);
Roku Proxy with SSL Support
Rokuality is the first platform to provide proxy support with ssl decryption for Roku devices! No need to tether your Roku to your machine via wifi sharing and no need to perform complicated router based modifications! See the following sections for details on proxy setup and usage.
To get started monitoring/modifying your Roku http(s) traffic during test, you must install one of our 2 supported proxies. Rokuality current supports Charles Proxy - Windows and Mac and Fiddler Classic. Both require some simple 1 time setup after you have installed the proxy of your choice.
Charles Proxy - First Time Setup
Install Charles Proxy version 4.5.6 or newer.
Launch Charles Proxy for the first time.
Optional - If you wish to run multiple instances of Charles Proxy while testing, i.e. test concurrency with each test having their own Charles Proxy instance, please see these instructions
This should be all that is required for Rokuality to be able to create Charles Proxy configurations and launch/stop/query the proxy as needed during automation.
Fiddler Classic - First Time Setup
Install the latest version of Fiddler Classic.
Launch Fiddler Classic for the first time.
Create a CustomRules.js file for the first time use by choosing the toolbar menu Rules>>Customize Rules.... This will create the backend CustomRules.js file that Rokuality needs in order to launch/stop/query the proxy during automation.
Allow remote connections from your device to the proxy by choosing the toolbar menu Tools>>Options>>Connections and check the Allow remote computers to connect checkbox.
Disable the system proxy on startup by choosing the toolbar menu Tools>>Options>>Connections and un-check the Act as a system proxy on startup checkbox.
Optional - if you wish to decrypt ssl traffic during test - Enable https connections by choosing the toolbar menu Tools>>Options>>HTTPS and check both the Capture HTTPS CONNECTs checkbox and the Decrypt HTTPS traffic from all processes checkbox. You may also wish to ensure that all protocols are entered for capture including <client>;ssl2;ssl3;tls1.0;tls1.1;tls1.2
Note that when using Fiddler - you may only have 1 instance of Fiddler running at one time, i.e. you may only run 1 test per machine at any one time. You can scale this by distributing your tests to multiple Rokuality servers but if you need concurrency, Charles Proxy might be a better solution for you.
Starting a Proxy
To start a proxy during test execution, you must provide the minimum required capabilities which include the ProxyHost and ProxyPort capabilities. See the capabilities section for additional capabilities. When the minimum caps for a Proxy connection are provided, a Charles or Fiddler proxy will be started on the backend and all traffic from your Roku device will then route through the proxy.
// Declare a new DeviceCapability object
DeviceCapabilities capabilities = new DeviceCapabilities();
// The ip address of your machine running Rokuality which allows your
// Roku to communicate with your Rokuality
// server and proxy server, i.e. 192.168.1.47
capabilities.addCapability("ProxyHost", "your server ip address");
// Any open port on your machine you wish the proxy to run on.
// Each test session must have a unique port
// Pass the capabilities and start the test
RokuDriver driver = new RokuDriver(":port", capabilities);
Roku Proxy: Monitoring http(s) Calls During Test
During the course of a test, the proxy api will allow you to retrieve your request/response traffic in standard har format. Several libraries exist that will allow you to parse your har content into objects during test, but at the core a har file is a json file that can be cast/queried as any JSONObject.
// gets the har file of every request in the proxy from test start to now
File harFile = driver.proxy().getHarLog();
System.out.println("Har file saved to: " + harFile);
// clear the har log