Time it takes for an indexer to respond to a search request

[rmorlen]

We are running searchhead pooling and have many indexers. I would like to be able to find out how long it takes for an indexer to respond to a search request. I do know that depending on how many events were indexed per indexer this number could vary.

I can get timing information by clicking the job inspector button and looking at the dispatch.stream.remote information (including the info for each indexer in dispatch.stream.remote.indexer1, dispatch.stream.remote.indexer2...).

Where in the internal logs can I find this? I can find references to the dispatch.stream.remote info but not for the individual indexers. I also didn't see the name of the search for that information.

I would like to create a scheduled search that runs every "x time" and compare the timing information for each indexer.

Any suggestions?

[lmyrefelt]

They are not in the internal indexes ( or so i believe 😉 ) , but rather they are in the $SPLUNK_HOME/var/run/splunk/dispatch/your_search_with_"cryptic"_name_dir/ in one of the logs, or csv there ( search.log ? )

You should be able to download this from the job-inspector .

[tmartin]

Try the REST API. The below should get you started on getting the info for a given search ID.

| rest /services/search/jobs | search sid="1397772637.152" |table sid,"performance.dispatch.stream.remote.*.duration_secs" | transpose

[rmorlen]

So if I have a scheduled search that is run at which point I would like to get the times for that search, how do I get the sid?