The What: Automated visual testing is a technique to verify that your application appears to the user as you intended. Visual testing can be used whenever there’s a need to verify the display of data on a web or mobile application. The Why: Traditional testing paradigms allow the creation of separate tests for their functional and visual checks. They utilize the functional tests to assert that all data is accurate, and they use the visual tests to assert that there are no visual bugs. While this is certainly a viable option, it is redundant. Not only is the visual check asserting that there are no visual bugs, but it is also asserting that everything on the page is correct, including the data. A Recent Visual Issue on the Shaadi.com platform has been a distortion of the bottom gradient in iPhones with screen sizes from 5.86 to 6.64 inches. The depiction of the issue has been highlighted in the screenshot below.
With the increasing number of digital channels that organizations have to deliver their content and features to their customers, It is becoming costlier by the day to verify the UI of each and every feature on each and every digital content delivery platform. It costs additional hardware, effort and time to the organization. Visual/UI Regression tools provide automated solutions for comparing the UI of our systems with the baseline or intended UI. The How: Just like functional assertions, visual assertions are used at the point of verification. An automation tool (e.g. Selenium, Cypress, Appium, etc.) is used to interact with the application and then a Visual Regression tool should be used to visually verify the outcome. Also, just like with functional assertions, it’s important to make the visual assertion only when the application is ready to be verified. For example, if your app needs to load data, your test code will need to wait until that data is loaded before verifying it. This is true of both functional and visual assertions. The Shaadi.com Challenge of Visual/UI Regression: The UI issue depicted above on the Sign-Up screen is a simple test case where any traditional UI regression tool can identify the difference by doing a pixelated comparison between intended and actual UIs as the page has static UI content. However, as we go deeper into our website and mobile apps, all our screens have a mixture of Static and dynamic UI elements like changing CTAs, texts, profile images, number of profiles etc. based upon the user who has logged into the system. The traditional approach of UI regression using pixelated comparison will give a huge number of false positives. Another challenge is to find a UI regression tool that is compatible to be executed on laptops/desktops/mobile phones/tablets based on different operating systems and then physically have that hardware to run the tests. Our Requirements:
- We need a smart UI Regression platform which provides us the level of flexibility as shown in the image below. We need to be able to configure the tool to be able to differentiate the sections of our screens which offer dynamic textual or graphic content. On top of this flexibility we need to accommodate the UI variations associated with different types of users like Premium, Free, Male, Female etc.
- The tool should provide its own device cloud so that we save the effort and complexity of integrating the tool with our regular testing device clouds.
Timelines: We have shortlisted 2 platforms namely applitools (Paid) and an Open Source Platform called WebDriver. IO – Visual Regression Service. We will be conducting POCs on both these platforms in the month of March 2021 and select one for implementation in Q1 & Q2 of 2021-22.
