Hi Simon,
Since both interfaces are working when they are primary, it looks to be good at network level.
Is that possible to share the bonding options used in your server?
Can you try switching the active slave using the command "ifenslave --change-active", instead of ifdown?
How about waiting for sometime to expire the Switch's forwarding table or ARP table?
I hope arp_interval and arp_ip_target also can help in refreshing the ARP table.
Thanks
Jibin Antony