We are using Splunk 6.5.6.
Recently we are seeing too many issue on alomst every server.
Is there any way all the server internal logs can be collected at cluster one place( it is costly and time consuming to configure email client on individual cluster member)
from that cluster server we can configure querry like "index=_internal error*" and mail this querry to our entire team. We can find and fix the issue soonest the possible.
Please suggest.
That's what your monitoring console is for: http://docs.splunk.com/Documentation/Splunk/7.1.0/DMC/Monitoringoverview
All Splunk instances send their logs to the indexer cluster, and the MC searches your cluster --> the MC can search logs from all Splunk instances.
If you have no MC set up, set up an MC.
If for some reason you cannot set up an MC, you can schedule alerts from your search head - it too can search the indexer cluster and therefore search all Splunk instances' logs.