Monitoring Splunk

Are older indexer servers slowing down overall search performance?

chris
Motivator

Hi,

we recently installed new indexer servers to help the existing indexer servers that were under heavy load. The new servers are newer and have more cpus/ram. It might not be a good idea to mix hardware, but that is the situation I have to deal with. The old indexers still have a higher load when I log in to them and check with top or sar. I am suspecting that they might also be the reason searches are still somewhat delayed. Is there an easy way to visualize how much time is spent for a search on each indexer? In the Job Inspector I can see how much time is spent for different parts of a search but not how much time a search spends on an indexer.

Regards
Chris

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

For a rough idea, check the dispatch.fetch region in the timing towards the top of the job inspector. You should see a figure for each search peer.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

For a rough idea, check the dispatch.fetch region in the timing towards the top of the job inspector. You should see a figure for each search peer.

View solution in original post

chris
Motivator

martin_mueller is right the Job inspector does show stats in dispatch.stream.remote: Job inspector
If always the same (old) servers are shown first this would either indicate that they are the only ones that have data or that searching takes longer.
The visualization is based on search.log of the Job which is not indexed (by Default).

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If you're looking for indexed data, newer versions of Splunk should also have a remote_searches.log (or similar) in _internal, that may or may not contain clues about this.

0 Karma

nkwong_splunk
Splunk Employee
Splunk Employee

If you are running Splunk Enterprise 6.1 or later on your indexers/search heads, the easiest way to view the health of all your Splunk instances is with the Distributed Management Console (DMC). The DMC provides out-of-the-box dashboards for search activity, indexing performance, resource usage, etc. for all the Splunk instances.

Here are additional details for the Search Activity dashboard within the DMC.
http://docs.splunk.com/Documentation/Splunk/6.3.2/DMC/SearchactivityInstance

Here are the setup steps for a multi-instance deployment of the DMC.
http://docs.splunk.com/Documentation/Splunk/6.3.2/DMC/Deploymentsetupsteps

0 Karma

chris
Motivator

Thanks for replying. The DMC is great to get a an overview. Unless I did not look properly the Search Activity dashboard does not show how much time is spent per indexer for a particular search.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.