A screenshot-based testing software

I just happened to know about Sikuli in this morning. Sikuli is a new automation framework, just like every others, but screenshot-based. The project is developed by MIT UID group. The project interests me enough to pay nearly the whole morning to play with it, and here are a few things I would like to note about this program. Before reading futher: I will only mention about the cons of the application, and you will have to find the pros yourself, check the blog for very cooooool examples of Sikuli first.

The 1st experiment with Sikuli: it's not very easy to write the script. My example is to test 1 + 1 = 2 on Calculator. So what I have to do is: type the script, take the screenshot, type the script, take the screenshot, type the script, take the screenshot, type the screenshot, take the script ... I mean I have to switch between the application and calculator all the time, and it makes me lose focus on what I am doing. And I wish taking the screenshot of the control is faster, i have to select the region of the button or control, why don't they implement in such way that you can select the control just on click ? I knew it's going to be a lot of work, since you will have to use native API to determine the controls, and you will have to maintain the integration between native API and the java application itself , but if we have this feature, thing will be 1000 times easier, or 2,3 times easier. It's just a suggestion.

The next thing is: it is very slow, or maybe I'm using XWGA screens with slow CPU. Or maybe because it's java based. The script is about 10 lines, with 5 pictures, but it takes about 3-5s to finish. Maybe it's the first beta version, we will see more performance improvement in later versions.

The third thing in the list is it's not correct for me sometime. The script halted, and I had to change the parameters to make it recognize the region. It would be better if we can limited the testing image to an application, in that way, we can improve the image recolonization, and the performance too. Right now, Sikuli keeps mistake the input of calculator with the explorer address bar.

[inline:sikuli-tweet.png]
Tweeting with Sikuli

And the main problem makes Sikuli is not suitable for application testing is the application itself. Sikuli script is not reusable, once you write the script, it's tighted with the environment of the OS you used to write the script. You won't be able to use the same script to test same application across computers, with different OS, or even just with different configuration. If you write the script for a web application, you will have problem when running the script with different browsers: Chrome and Firefox for example. If you happen to use different theme for your computer, the chance the script won't run is very high (I test it with Classic XP and XP with default theme). It's not all, you won't be able to reuse the script with script with different input, i.e the same script but with different values. You can use functions for repeating steps, but I had a big problem with evaluating the output. Or I have to capture all screenshots for all output ? Or maybe we should implement OCR to read the value from a textbox so we can evaluate the result. But it's not possible to have that right now.

I must say they have done a great job, this is going to be a very promising project, and I really hope to be able to use Sikuli to automate my work, such as logging work. Maybe you will have a better idea ?