Remote Buddy is actually very quick in triggering actions (like sending a UDP packet to XBMC) in reaction to button presses and has a unique end-to-end-optimization that already begins at the raw data coming in from the receiver and also takes your individual mapping settings into account to avoid any avoidable delays. For the Apple IR Receiver, to maximize the benefit of this optimization, you'll need to install Remote Buddy's own driver (Candelair), though: the stock OS X Apple Remote driver - for many buttons - introduces delays of up to half a second before creating any event applications can receive. With that driver, it's also impossible for applications to reliably detect the begin and end of a button press (which, too, can introduce delays). That's only possible with Candelair.

Regarding your performance issue, since this is a very complex topic, please provide me with more information:

1) your Mac model and how much memory it has

2) which other applications you're running at the same time you're running XBMC and Remote Buddy

3) your settings for your Apple Remote at Prefs > Mapping > Global Mapping and Prefs > Mapping > XBMC (for simplicity, you may consider just making a screenshot and posting it on f.ex. droplr.com, which doesn't require any signup)

4) whether you have installed Remote Buddy's driver (Candelair) - or not

5) which Apple Remote model you have (white plastic or aluminum)

6) whether you've already tried with a new battery (the Apple Remote will send an extra IR code before any button press when the battery level is low - which introduces a delay and can even lead to long button presses becoming virtually impossible to make)

Thanks in advance.

Best regards, 
Felix Schwarz