Why is searching data in warm buckets extremely slow after 50 to 60 percent of the search has been completed?

[Phentermime]

Hi All,

We're having an issue with our Splunk where data older than approximately 8 days are extremely slow to search.

First part of the search is done in a few seconds and Splunk says it's between 50 to 60 percent done with the search. The last 40 to 50 percent of the search takes several minutes to complete.

An inspection of the job tells us that the search, amongst others, spends a healthy amount of time in command.search.filter with command.search.rawdata and command.search.kv. But it is nowhere near as much time as the the whole search uses. The search (from a 3rd party app) is using the tstat command and this is noted in the inspection as taking 400 to 500 seconds to complete.
The app uses datamodels and the inspection never mentions anything about the search being done in indexes.

Anyone having any ideas why this is so slow?

We've got one big physical server as an indexer and search head as well as two other search heads along side.

We can't see anything that could indicate a performance issue on the RHEL server that could explain this behaviour.

Any suggestions would help.

Thanks,
John

[Phentermime · Accepted solution]

Hi all,

Such utter stupidity on my part. And I really can't blame it on anyone else!

This is of course an app that uses data models with acceleration and it was not set correctly for our use. To change it all that's needed is to select "Edit Acceleration" in the Data Models screen for the app's data model (or in the app's datamodels.conf file), adjust the settings and give it good amount of time to re-build and of course make sure there's enough disk space.

Rgds,
John.

[aaraneta_splunk]

Hi @Phentermime - You mentioned that a third-party app is being used during search. I'm wondering, are you using a Splunk-based app downloaded from Splunkbase? https://splunkbase.splunk.com/

Or is it a third-party app developed outside of Splunk?

The reason I ask is because if it's a Splunk-based app or add-on, I want to make sure I include that specific tag in your post so that we can get more visibility for you. Thanks.

[Phentermime]

Hi,

it's on Splunkbase. UberAgent.

Rgds,
John