Prev/Next probably work because the commands send to Front Row are equivalent to the commands sent to iTunes. Play/Pause actually restarts the track, and none of the other buttons do anything, except for holding down the Menu button, which of course brings up the RB overlay menu, by which I can then exit Front Row by selecting the "Quick Exit" option (and in fact this is about the only way I can out of Front Row at this point, short of doing a CMD-OPT-ESC operation).

The problem is that the only way the custom action can be made to work is to direct the CMD-T and CMD-F keystrokes specifically at iTunes and set it to activate iTunes as the application. Otherwise, the options don't do anything unless iTunes is already active.

I'm going to play with it a bit more and see if I can't create another custom action that will reactivate Front Row when shutting down the visualizer.