Are older indexer servers slowing down overall search performance?

[chris]

Hi,

we recently installed new indexer servers to help the existing indexer servers that were under heavy load. The new servers are newer and have more cpus/ram. It might not be a good idea to mix hardware, but that is the situation I have to deal with. The old indexers still have a higher load when I log in to them and check with top or sar. I am suspecting that they might also be the reason searches are still somewhat delayed. Is there an easy way to visualize how much time is spent for a search on each indexer? In the Job Inspector I can see how much time is spent for different parts of a search but not how much time a search spends on an indexer.

Regards
Chris

[martin_mueller · Accepted solution]

For a rough idea, check the dispatch.fetch region in the timing towards the top of the job inspector. You should see a figure for each search peer.

[chris]

martin_mueller is right the Job inspector does show stats in dispatch.stream.remote:
If always the same (old) servers are shown first this would either indicate that they are the only ones that have data or that searching takes longer.
The visualization is based on search.log of the Job which is not indexed (by Default).

[martin_mueller]

If you're looking for indexed data, newer versions of Splunk should also have a remote_searches.log (or similar) in _internal, that may or may not contain clues about this.

[nkwong_splunk]

If you are running Splunk Enterprise 6.1 or later on your indexers/search heads, the easiest way to view the health of all your Splunk instances is with the Distributed Management Console (DMC). The DMC provides out-of-the-box dashboards for search activity, indexing performance, resource usage, etc. for all the Splunk instances.

Here are additional details for the Search Activity dashboard within the DMC.
http://docs.splunk.com/Documentation/Splunk/6.3.2/DMC/SearchactivityInstance

Here are the setup steps for a multi-instance deployment of the DMC.
http://docs.splunk.com/Documentation/Splunk/6.3.2/DMC/Deploymentsetupsteps

[chris]

Thanks for replying. The DMC is great to get a an overview. Unless I did not look properly the Search Activity dashboard does not show how much time is spent per indexer for a particular search.