Skip to Content

Sikuli can automate any UI by taking screenshots

This is pretty impressive --of course there are already lots of ways to automate actions on your Mac, and odds are that you may have messed around with scripting or Automator more than once before. But Sikuli is a new app that makes automating as simple as taking screenshots. Instead of programming actions, all you have to do is put screenshots of the particular UI items that you want to automate in a list, and then the app will use visual cues to do whatever you want it to do. Not only does it make things much easier when actually writing scripts, but it also gives you countless options in terms of automation -- the app can automate any app that has a graphical user interface, because all it has to do is recognize that GUI on your screen. That includes web apps, too -- like I said, the possibilities are endless.

And since it's developed with Jython, you Python experts can insert any Python code that you like in the scripts as well. The whole thing is an MIT project, so it's available for free across all platforms, including OS X, Windows, and Linux. If you've been looking for an automator that'll do any rote task for you, give Sikuli a try.

[via LifeHacker]

Categories

OS Software Freeware

This is pretty impressive --of course there are already lots of ways to automate actions on your Mac, and odds are that you may have messed...
 

Add a Comment

*0 / 3000 Character Maximum

11 Comments

Filter by:
jstark

Look interesting. I'd be shocked if it was as feature rich as other similar products like QuicKeys or Eggplant. Definitely worth a test drive. Thanks for the heads up.

February 03 2010 at 8:40 PM Report abuse rate up rate down Reply
Nitin Pande

I think it is still evolving and will take time before it can become very fast.
I too have compiled an article bringing together all available tutorials and videos.

http://technoticles.com/2010/01/23/sikuli-the-graphical-scripting-language-from-mit/

February 02 2010 at 5:06 PM Report abuse rate up rate down Reply
mech

this thing is PERFECT! exactly what I've been looking for! Thanks TUAW. I saw this on LH, but didn't realize it's full potential. Thanks for showing the demo vid.

I finally figured out how to use iAlertU and Proximity to setup security on my OSX netbook. iAlertU doesn't have a library of APIs accessible for scripting, so this visual scripting is what I needed to make it all work.

Now, when I leave the proximity of my netbook in a coffee shop, Proximity activates the 'outofrange' script, which has a 'do shell script "open ~/Dropbox/scripts/sikuli_iAlertU.skl" ', which is an executable created in Sikuli.
The Sikuli executable was easily coded using Sikuli and showing iAlertU in the top menu bar for easy clicking.

The whole process arms iAlertU within ~5 seconds. When I come back, Proximity runs 'inrange' script, which has a simple 'keystroke "password" ' script. Not the best security, but given iAlertU's effectiveness for drawing attention, no one would dare take my netbook without drawing lots of attention.

I'm reworking my inrange script to access the keychain so as not to explicitly type it into a script, but this is for another time.

Thanks guys :)

February 02 2010 at 4:46 PM Report abuse rate up rate down Reply
Jars

A Tedious work that can be done by changing/creating your location, In less steps.

Apart from that, interesting. I can see more people using this easy tool to automate other tedious tasks knowing only how to use the UI.

February 02 2010 at 5:44 AM Report abuse rate up rate down Reply
Mike

it looks like it takes a while to find the button from the demo… idk if lll use this or not..

February 01 2010 at 11:52 PM Report abuse rate up rate down Reply
anonymous

I tried this. I promise, it works amazingly, except it can get a little slow.

February 01 2010 at 10:18 PM Report abuse rate up rate down Reply
Bagster

I had to use a product like that before, called Citratest. I will consider using Sikuli instead, because the support cost of citratest is really high...

February 01 2010 at 9:33 PM Report abuse rate up rate down Reply
MacSlut

Wakka,

This is different because there doesn't have to be any hooks in the app for it to work. AppleScript/Automator can work only with what a developer allows. There's a "Macslut's Law" that says anything you want a script to do the developer of the app has not enabled.

This overcomes this problem. If this app sees a button and you tell it to push it, it will regardless of whether or not a developer has enabled that as an action. Sikuli just needs to see it happen.

February 01 2010 at 8:57 PM Report abuse rate up rate down Reply
mac

I tried many times to make a macro/script in automator to delete my call logs in skype and again in adium - with one click. It wouldnt work. I lost interest.
Might try this but if it gives me a headache too, I'll just delete them manually.

February 01 2010 at 7:40 PM Report abuse rate up rate down Reply
BenDem

How is this any different from recording a macro in Automator?

February 01 2010 at 7:35 PM Report abuse rate up rate down Reply
1 reply to BenDem's comment
Murphy Mac

This isn't limited by APIs, AppleScript, or Automator capabilities. It IS probably limited by what you can do in a GUI -- from what I read in the linked post. It sounds like the app tries to recognize buttons you clicked and records keystrokes and maybe keyboard shortcuts you invoke. Then it tries to immitate your actions when it runs.

You should read the Lifehacker post Mike linked to.

February 01 2010 at 8:55 PM Report abuse rate up rate down Reply
Buy an ad here

Hot Apps on TUAW

Tweets

© 2012 AOL Inc. All Rights Reserved.