This is really more of a discussion topic than an actual question. Definitely something more suited to the IRC channel!
That said however, I will provide some URL's to Docu/Wiki pages that help to build a bigger picture. It should be stressed that deployments can vary massively and its quite hard to cover all bases before going in. Not to mention if you are after system specific issues it needs to be remembered that Splunk operates across alot of different network / hardware configurations and it's unlikely that you'll get a coherent response that is useful to yourself.