I have a working environment using index discovery.
While doing command 'tail -f' on idx01 (Indexer - master box), I notice the following logs.
10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx03 at uri https://10.200.2.35:8089 because replication was unsuccessful. replicationStatus Failed failure info: failedbecauseHTTPREPLYREADFAILURE
10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx04 at uri https://10.200.2.38:8089 because replication was unsuccessful. replicationStatus Failed failure info: failedbecauseHTTPCONNECTIONFAILURE
10-19-2016 20:59:38.186 +0000 INFO TcpOutputProc - Connected to idx=10.200.2.36:9997
10-19-2016 20:59:46.316 +0000 WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 3 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsedms=19129, tarelapsedms=3726, bundlefilesize=75900KB, replicationid=1476910767, replicationreason="async replication allowed"
I did search using this string 'Failed failure info' on splunk, but found nothing.
I am not sure if logs on indexers are parsed and indexed.
What is the best practice to monitor/analyze logs on indexers/search head boxes?
the splunkd.log on your indexers should be indexed automatically. You can find them by searching index=_internal. There you will find all internal splunk logdata.
For getting the internal logs of your searchhead to the indexer tier look at this. This works similar for your master node.
I did not use 'index=internal' on earlier attempt.
It works by entering the following search, using 'index=internal'
index="_internal" Failed failure info source="/opt/splunk/var/log/splunk/splunkd.log"
Thank you @TStrauch