topic Re: Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer in Deployment Architecture

Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer

hartfoml — Wed, 24 Aug 2016 12:46:38 GMT

I have three indexers. All configured the same all with the same hardware (16 cores 32 GB ram).
I have a simple search for internal data
index=_internal host=My-License-Manager source=*license_usage.log type="RolloverSummary" earliest=-30d@d
This search runs in just over 5 seconds on indexer #1 and times out on indexer #2 and #3
If I change the time to earliest=-35d@d latest=-4d@d indexer #2 returns in 5 seconds but only #3 times out.
If I change the time to earliest=-29d@d latest=-4d@d all three indexers return results in just over 5 seconds.
One day later or one day earlier will cause indexer #2 or #3 to time out.

how do I start to troubleshoot what is causing this. I am sure this can't be isolated to this one data set and has to be affecting other data sets as well.

I opened a Case Number 387826 Date/Time Opened 8/23/2016 7:31 AM with splunk support but no response yet

Re: Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer

gfuente — Tue, 29 Sep 2020 10:43:17 GMT

Hello

Maybe your data is not properly balanced, if for some reason the _internal data of a few days it´s contanined only in indexer #2, then it´s going to take much longer to retrieve the events for those days.

I will recomend creating a timechart with the count of the number of events per indexder, using:

index=_internal host=My-License-Manager source=*license_usage.log type="RolloverSummary" | timechart count by splunk_server

If you have issues, try filtering one splunk_server at a time, and compare results

Regards

Re: Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer

hartfoml — Wed, 24 Aug 2016 13:04:57 GMT

@gfuente I thing you and I are thinking alike. I did the very same troubleshooting steps you suggested in fact the LicenseUsage - type=RolloverSummary logs for each day only show up on one indexer per day. when I do the search with the timeframe earliest=-29d@d latest=-4d@d I get

1 (139) events

2 (68) events

3 (136) events

When I do the search per day by indexer

1 has 10 days

2 has 5 days

3 has 10 days

The data is not evenly balanced but when I do the earliest=-29d@d latest=-4d@d the search returned fast with a dispatch.fetch time of just over 7 seconds. If I change the day by one, later or earlier, one of the search peers times out.

Thanks for the suggestion

Re: Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer

hartfoml — Wed, 24 Aug 2016 15:44:46 GMT

Sorry I called support and they said Skip would take the case. That probably means it will be answered by Skip as he is the (cats meow) at splunk support.

If your going to get point for this you have to hurry cause Skip is on the case...