Monitoring Splunk

Why does an initial pivot search take so long?

aweitzman
Motivator

Using Splunk 6.1.4 (running on Windows, if that matters), I have created a fairly simple data model with one simple object in it, and since I want the associated pivot table to appear as a menu item in an app, I have saved it as a saved search as described here: http://answers.splunk.com/answers/109375/datamodel-and-pivot-export-with-app.html . When I go to the pivot table from the app menu, it pulls up the right thing, with the time restriction I want (Last 24 hours) properly filled in.

However, the initial search takes forever! When I do just the search that the pivot table is based on over the last 24 hours from the Search bar, it returns results in 16 seconds. When I pull up the pivot table version for the last 24 hours, it takes almost 11 minutes!

So I looked in the Jobs table, and it seems that it's attempting to pull in way more data than it needs to, because the "earliest" time for the pivot table search is not 24 hours ago, but almost three years ago, basically the equivalent of All time! This would explain the very long time for the search, but it doesn't explain why it's doing this in the first place. Why is it going back through all the data for a 24-hour search?

(Note: It also does this when I go to the pivot table from the Pivot menu in the Search app, and then change the time to Last 24 hours.)

It always looks like it gets some percentage of the way through the search fairly quickly (in this case, about 65%) and then appears to halt for the remainder of the 11 minute search time, with almost no indication that it's doing anything.

So, what is going on? How do I get the initial pivot search to use the "earliest" time I give it for the search, so that it doesn't take so long to complete?

0 Karma
1 Solution

aweitzman
Motivator

I think I've solved this problem by accelerating the data model. Not only does this make most of the searches faster generally, but it seems to short-circuit the "search everything" process and allow it to only search the data within the timeframe of the search, even when it's outside the boundaries of the acceleration.

Seems curious that it would behave that way, but there you go.

View solution in original post

aweitzman
Motivator

I think I've solved this problem by accelerating the data model. Not only does this make most of the searches faster generally, but it seems to short-circuit the "search everything" process and allow it to only search the data within the timeframe of the search, even when it's outside the boundaries of the acceleration.

Seems curious that it would behave that way, but there you go.

dlamas_splunk
Splunk Employee
Splunk Employee

Is "pivot_adhoc_acceleration_mode" in web.conf set to "Elastic?" This is the default, but you may have a conflicting setting in your local web.conf.

aweitzman
Motivator

Yes, it is explicitly set to "Elastic" in the default web.conf, and not overridden anywhere.

Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...