I know it has been a year but I figured I would respond to this. So many people I know buy a Zigbee (series 2) XBee thinking it will be better, but the Zigbee protocol is really designed for a certain special niche where you need low low power end nodes and very few broadcast transmissions. To get a series 2 to talk like you want you would need to change the DL and DH on each radio to match the SL and SH of the radio you want it to send to. (you could also use API mode to send a DL and DH with each packet if your controller has the smarts to do so.). Sending with no destination is a broadcast and every time you try to broadcast the network will wait and wait for any possible responses. If it instead knows the destination it is a nice fast simple send and reply.
Or you could get a series 1 radio like you mention, they run point to point (802.15.4) or Digimesh either of which would run a ton faster in this situation.