User

30.08.2006 23:33:30

Felix,

I would propose you to reconsider the current, tedious way of entering actions step-by-step in Behaviour Factory. With UI scripting, RemoteBuddy could "read" a front application's available menu's and GUI elements, present those onscreen in its own format, and thus control apps exactly as the Mac UI does. 
As known, UI scripting is not ALWAYS without problems but in these cases step-by-step action editing can come in.

Please find enclosed the following script that illustrates how to interactively "read" UI elements and emulate selections - as RB could do. 
I would suggest to introduce such an option into RB's initial Behaviour dialog. 
This will certainly facilitate interfacing with always new and updated applications on our disks !

Regards, 
Eelco Houwink

_________

tell application "System Events" 
set fApp to name of some application process whose frontmost is true and background only is false 
set Appz to name of application processes 
tell application fApp 
activate 
set appn to choose from list Appz with prompt "Select app" OK button name "OK" 
end tell 
if appn is false then return 
set targName to item 1 of appn 
set theApp to targName 
set mainTarget to application process targName

repeat 
set fApp to name of some application process whose frontmost is true 
tell process fApp to activate 
set UIlist to (get mainTarget's UI elements) 
set UInames to {} 
repeat with h from 1 to number of items in UIlist 
try 
set name1 to (get name of item h of UIlist) 
if name1 = "" then set name1 to ("itempje " & h) as string 
on error 
set name1 to (class of item h of UIlist & " " & h) as string 
end try 
set UInames to UInames & {name1} 
end repeat 
if UInames = {} then exit repeat 
if number of items of UInames = 1 then 
set selname to item 1 of UInames 
else 
set selname to (choose from list UInames with prompt "UI elements of \"" & targName & "\": " default items 1 OK button name "Browse") as string 
if selname is "false" then exit repeat 
end if 
set itemNo to my getIndex(selname, UInames) 
set mainTarget to item itemNo of UIlist 
try 
set targName to name of mainTarget 
if name of mainTarget = "" then set targName to fApp 
on error 
set targName to fApp 
end try 
end repeat 
end tell 
repeat until fApp = theApp 
tell application theApp to activate 
tell me to do shell script "sleep 1" 
tell application "System Events" to set fApp to name of some application process whose frontmost is true 
end repeat 
tell application "System Events" to click mainTarget

on getIndex(i, l) 
repeat with n from 1 to count l 
if l's item n is i then return n 
end repeat 
0 
end getIndex

Last edited: 31.08.2006 00:16:12 

User

31.08.2006 20:07:02

Hello Eelco,

wow! Thanks a lot for that contribution!

I do see problems with it though, namely with localization, i.e. 
trying f.ex. to access the about menu of an app, any stored values 
would have to be different for different languages, i.e. "About this 
Mac" in English and "Über diesen Mac" in German .. but I see the use 
and power behind this and will elaborate further into that direction 
once I find the necessary time!

Again, thanks a lot!

Best regards, 
Felix 

User

01.09.2006 14:34:43

Felix,

Localisation isn't supposed to be an issue as it UI scripting should refer to names in their localised form (or can indentify ID's)

So, this is certainly a huge undertaking but still worthwile. 
It may save you from a lot of the Microsoft pecularities. 
Powerpoint UI-scripting reveals ALL it's menu options...

However, there are other issue's: 
Unknown characters and Menu separators do issue unusable references and, in full-screen mode, commands are not UI-elements and can't be queried this way.

So, the development strategy could be a mixed-mode between UI scripting and Behaviour construction. 
For example, UI scripting menu as a alternative menu option, where UI scripting errors will fall back into a Behaviour or a pop-up window to define an Action in that Behaviour.

It will be a challenge to NOT confuse the user with these pecularities...

Eelco

Last edited: 01.09.2006 14:49:40 

User

04.09.2006 23:07:02

Hello Eelco,

I spent some time thinking about it and I'll spend more time 
exploring all the possibilities in that direction after the release 
of preview 7. This really looks interesting, but also fragile (i.e. 
even with menu offsets, any update to the software in question is 
likely to break it - as are in the case of different versions that 
are in circulation - I did, however, not yet explore UI scripting to 
its full extent, so my judgement shall not be regarded final ;-).

It's my goal to keep the BCK and the "user" interface seperate from 
each other, so while I think a replacement action window would be 
great for the scenario of a broken UI script, it's probably nothing 
I'll do since I, as you, fear it'll confuse far more people than it 
will help and it'll also break the "seperation" of those two parts of 
the app.

Best regards, 
Felix 

User

04.09.2006 23:32:41

I see your point and fragility certainly is an issue with GUI scripting. 
However, I guess it would better cope with updated software than strict Behavior rules do. You may check that with different versions of e.g. Powerpoint. Unlike BCK, it is supposed to adapt to the latest version's GUI anyway.

Keeping BCK separate from the user interface should be optional: 
I would say that for debugging purposes a direct dig into the BCK hierarchy would be more preferable than isolation.

So: a practical combination of both worlds... 
(see various comments on Versiontracker on RemoteBuddy and competition)

Implementing 2 simple interface commands would provide scripters the essential hooks to RB: 
1. ShowMenu(list,insertlocation) -> Passing a list {"option 1","option 2","option 3"} somewhere in the RB menu hierarchy to display it onscreen. 
2. DidSelect(menu x, item y) -> returning selected option back to the calling AppleScript.

This way, it's up to the scripter to intercept GUI anomalies.

Last edited: 04.09.2006 23:53:46 

User

12.09.2006 01:40:02

Hi Eelco,

thanks for following up!

Regarding scripting and fragility: preview 7 continues to be just a 
milestone to the final 1.0 and some of the new features I plan may be 
very beneficial to the idea of using UI scripting in Remote Buddy. I 
still do, however, to find some time to really dig deep into this to 
get the full view of what's possible and what's not.

There'll certainly be *some* degree of integration between the BCK 
and the rest of RB in future. At least in places where it is a real 
help to Joe Average and does not get confusing. But I've yet to work 
out all of the technical and conceptional details so everything works 
nicely and seamlessly together.

Best regards, 
Felix