Integration of Selenium + Sikuli

Sikuli is a tool using Visual Recognition capability which automates anything seen on the screen of your computer running Windows, Mac, few Linux/Unix OS systems. It uses Image recognition powered by OpenCV to identify the GUI components.

The version of Sikuli is named as 'SikuliX'. It also comes with basic text recognition (OCR) feature to search text within images.Besides locating images on screen, it can handle mouse & keyboard events to interact with GUI elements. 


Sikuli for Android devices based on ADB(Android Debugging Bridge) is in experimental stage.

Sikuli along with Selenium make a strong combination which would eliminate many limitation of Selenium like Handling of native Pop-ups, Flash/Flex components, native UI components etc.

Set up for Java Programming with Sikuli:

1. Download the SikuliXapi.jar & related OS libs jar (sikuliXlibswin.jar).
2. Add the downloaded libraries as eclipse project dependencies.
3. Incase of error like 'NoClassDefFoundError org/bridj during execution, Add "bridj-0.*.*.jar" into project dependencies.

Sample code:

The below code would let us know how selenium & sikuli can be integrated.

Scenario: Google Image Search

public class SikuliTest {

static String imageFolder = System.getProperty("user.dir")+"/ReferenceImages/";
static String chromeDriverPath = System.getProperty("user.dir") + "/libs/chromedriver.exe";

public static void main(String[] args) throws InterruptedException {

System.setProperty("webdriver.chrome.driver",chromeDriverPath);
WebDriver driver = new ChromeDriver();

//Set ImagePath & Create instance of Sikuli
ImagePath.setBundlePath(imageFolder);
Screen scr = new Screen();

try {
driver.get("https://images.google.co.in/");
driver.manage().window().maximize();
Thread.sleep(3000);
scr.click("CameraIcon.png");

driver.findElement(By.xpath("//div/a[contains(text(),'Upload an image')]")).click();
driver.findElement(By.xpath("//input[@id='qbfile']")).click();

scr.type("FilePath.png","C:\\PathTo\\S_logo.jpg");
scr.click("open.png");
//Wait for Image to appear
scr.wait("SearchPageHeader.png", 6000);
//Capture the screenshot of searchResults Page
scr.saveScreenCapture("C:\\Images\\","SrchRslt.jpg");

driver.close();
driver.quit();
} catch (FindFailed e) {
e.printStackTrace();
}
}

}

Note & References:

• Images captured out of Sikuli IDE can also be used as reference images.
• http://sikulix.com/quickstart/
• https://github.com/RaiMan/SikuliX-2014/wiki/Usage-in-Java-programming