14

I would like to use Sikuli to automate both GUI apps and Web apps running within browser on Mac OS X and Windows. My purpose is currently less for testing, and more for GUI automation of tedious, repetitive tasks for a team that unfortunately doesn't have lower-level automation access at this time.

I'm thinking that I'd like to build up one or more libraries of screen shots for the GUI apps and Web apps that I can reuse across projects. I'd often be running the same automation steps for different apps or, for Web apps, in different browser/platform combinations.

What are some good strategies for constructing reusable Sikuli screen shot libraries? Some thoughts:

  • should I capture screen shots outside of Sikuli, and then slice/dice those images to pull out specific interface elements within Sikuli?
  • how can I best keep track of screen shots for equivalent interface elements across similar GUI apps?
  • how can I best keep track of screen shots the same Web apps as seen in different browsers or platforms?
  • how can I best organize elements that are hierarchical, like menus where you must make choice 1, then choice 2, then choice 3 (but the next choice only appears after the previous one is selected)?
  • should screen shots be saved as variables to be able to reference them more generically?
  • should I construct Python lists or dictionaries that contain screen shots?
  • should I group screen shots into separate Sikuli files based on application/platform?

I'm assuming in all of this that I could import the libraries like a Python module, which certainly seems possible from the documentation.

Thanks!

5 Answers 5

18

There is an add-on called "Robust GUI Automation Library for Sikuli".

Even if you don't end up using the library, there are some really good lessons to be learned by looking at their implementation of the problem.

A few suggestions:

should I capture screen shots outside of Sikuli, and then slice/dice those images to pull out specific interface elements within Sikuli?

  1. More important than how you get your elements is how those elements are stored. I standardize how I name graphics ie: Button_OK.png rather than Sikuli's unpredictable_default_name.png

  2. You can add image libraries "on the fly" in your Sikuli script. Store different browser and platform graphics in different directories.

    myImagePath = "M:\\myImageLibrary\\"
    addImagePath(myImagePath)
    

how can I best keep track of screen shots for equivalent interface elements across similar GUI apps?

Naming conventions!

\\firefox\\Button_OK.png
\\IE8\\Button_OK.png

You can also play with the "similarity" of the Pattern to get the same graphic to hit on both IE and Firefox (but without false positives). This can take a bit of trial and error.

should I construct Python lists or dictionaries that contain screen shots?

This is a really good practice and has worked well for me in certain circumstances. Sometimes though, the filename is better documentation of the script functionality than a list offset.

I'm assuming in all of this that I could import the libraries like a Python module, which certainly seems possible from the documentation.

Yes you can import libraries.

5

That looks like a great library recommended by spearson.

I would add one more concept to the list, which is calibration.

As with any testing industries, calibration of your instruments is a must.

Within the SQA/automation fields, assumption can lead to disaster.

Scenario:

On Monday, you decide on your Chrome submit button screen shots to be used in Sikuli-powered automation.

You work fast and by Tuesday, your test suite is delivering accurate pass/fails as you expect.

On Friday afternoon, just before beer'o'clock, the machine auto-upgrades to the next minor release of Chrome, which modifies the cancel buttons just enough to be matched by Sikuli as submit buttons.

You glance at your reports before leaving the office for the weekend and your reports seem to be running fine as usual, but you don't realize they're giving false-positives until dreaded Monday when things have been broken all weekend (but, hey, at least you had a good weekend!).

Totally hypothetical situation, but hopefully stresses the need for "testing your tests" or calibrating your tools in a write-once, run-many automation environment.

Solution:

To mitigate problematic situations like the one above, you could setup a web page(s) that you know behaves a certain way to interactions with the screenshots in your static library. Before each test suite/bulk automation project runs, it will call the calibration suite and make sure everything is functioning as expected, be it a browser, file manager, etc

0

As a Java developer (not sure if this applies to many sikuli users): I rolled my own implementation. I created a singleton FileDirectory object which has methods for default directories (resources, patterns, screenshots). And whenever I had sikuli take a screenshot I would save it to this default directory (which was created if it didn't exist on start-up). My application would than log the screenshot. Using an html logger I could link to the screenshot directory. So my html logs had links embeded in them to the images the application took.

I know this might seem difficult at first, but it ended up providing exactly what use case needed.

0

Maybe this will give you some idea's.
I have a file where I have multiple definition that need to look for a certain image.
Somethimes the image they need to look for is different in different browsers.
So I use something like this:

File: ImageLib.sikuli

browser_Windows_Firefox = ("windowsFox.png")
browser_Mac_Firefox = ("macFox.png")

File: Execute.sikuli

from ImageLib import *

# Variable you can get set somewhere 
operatingSystem = 'Mac'

image = ()
if (operatingSystem == 'Windows'):
    image = browser_Windows_Firefox
elif (operatingSystem == 'Mac'):
    image = browser_Mac_Firefox
else:
    # Unknow option here.... 
    pass

# Find and highlight it. 
imageLoc = find(image)
imageLoc.highlight(5)
0

Jaharmi, this is a great question that still has relevance 10 years later. Writing complex automation tends to get messy quickly. It’s my own project, but I’d like to recommend Brobot, an open-source automation framework using SikuliX and OpenCV that happens to address all your questions directly.

how can I best organize elements that are hierarchical

Brobot builds a state structure, which is comprised of states and transitions. States hold only objects (no methods) and transitions provide instructions to get to other states. In the image below, which could correspond to your example of a hierarchical menu, squares represent states and circles represent transitions. Brobot manages all movement from one state to another through path finding algorithms and the user-defined transitions. You can then move to another state with one line of code: stateTransitionsManagement.open(STATE_TO_OPEN);.

hierarchical process flow

how can I best keep track of screen shots for equivalent interface elements across similar GUI apps?

Image objects in Brobot can contain multiple image files. You can have a single Brobot Image with an image file for each GUI app.

Several more questions ask about screenshots. Screenshots are important as a representation of your target environment and should be saved. They will not be used in code and do not need to exist as variables but will be used initially for image analysis and can be used later for unit tests. This is the role screenshots play in creating a state structure:

  • Brobot saves screenshots while you control manually the target environment. All screenshots live in one folder and represent a process flow.
  • You select images from these screenshots you wish to use in your application.
  • Brobot performs an analysis of your selected images on your screenshots.
  • Brobot builds the state structure by writing Java code in your project directory, creating folders with state and transitions classes. Your images are now organized into states and contain ‘historical matches’ from your screenshot library that can be used for simulating your environment and testing your codebase.

There is a video tutorial series on how to build the state structure.

The Brobot introductory video explains the concept of states at 3:39. The documentation also has a page on states.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.