Our Sales Engineer said -
-- The best practice (not just for Splunk, really for the whole industry) is to put your UDP receiver as close as possible to the sender. Like, on the same switch if you can help it. This will minimize the lost packets that are indeed inevitable with UDP.
The other thing you should look into is whether UDP is really required. Most appliances / switches / firewalls / routers these days have an alternative way to send data. Usually an API, TCP bound syslog, a form of reliable/resilient UDP etc. Literally anything is better than basic UDP.