Deployment Architecture

Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer

hartfoml
Motivator

I have three indexers. All configured the same all with the same hardware (16 cores 32 GB ram).
I have a simple search for internal data
index=_internal host=My-License-Manager source=*license_usage.log type="RolloverSummary" earliest=-30d@d
This search runs in just over 5 seconds on indexer #1 and times out on indexer #2 and #3
If I change the time to earliest=-35d@d latest=-4d@d indexer #2 returns in 5 seconds but only #3 times out.
If I change the time to earliest=-29d@d latest=-4d@d all three indexers return results in just over 5 seconds.
One day later or one day earlier will cause indexer #2 or #3 to time out.

how do I start to troubleshoot what is causing this. I am sure this can't be isolated to this one data set and has to be affecting other data sets as well.

I opened a Case Number 387826 Date/Time Opened 8/23/2016 7:31 AM with splunk support but no response yet

0 Karma

hartfoml
Motivator

Sorry I called support and they said Skip would take the case. That probably means it will be answered by Skip as he is the (cats meow) at splunk support.

If your going to get point for this you have to hurry cause Skip is on the case...

0 Karma

gfuente
Motivator

Hello

Maybe your data is not properly balanced, if for some reason the _internal data of a few days it´s contanined only in indexer #2, then it´s going to take much longer to retrieve the events for those days.

I will recomend creating a timechart with the count of the number of events per indexder, using:

index=_internal host=My-License-Manager source=*license_usage.log type="RolloverSummary" | timechart count by splunk_server

If you have issues, try filtering one splunk_server at a time, and compare results

Regards

0 Karma

hartfoml
Motivator

@gfuente I thing you and I are thinking alike. I did the very same troubleshooting steps you suggested in fact the LicenseUsage - type=RolloverSummary logs for each day only show up on one indexer per day. when I do the search with the timeframe earliest=-29d@d latest=-4d@d I get

1 (139) events

2 (68) events

3 (136) events

When I do the search per day by indexer

1 has 10 days

2 has 5 days

3 has 10 days

The data is not evenly balanced but when I do the earliest=-29d@d latest=-4d@d the search returned fast with a dispatch.fetch time of just over 7 seconds. If I change the day by one, later or earlier, one of the search peers times out.

Thanks for the suggestion

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...