When I ran a search spanning an entire year it took 241 seconds. If I immediately rerun the search the time plummets to ~60 seconds. Why? Is this a Splunk or Disk optimization? Background: hot/warm sit on fast disk. coldlib resides on not as fast, bigger disk. Regardless of the search I run, when the data is polled the first time it's always a slower reply. When the I rerun the same exact search over the same exact disks the times drop considerably. Who's responsible? (who can I thank?) Splunk or Disks....and is it that easy, or is it more complex? I understand that searching back onto colddb disks will require a slower retrieval vs. warm/hotdb. The question is more of a lower level, backend one. But one I want to share with my user base when I advise them how to tune their searches and what will happen when they rerun the search. I've looked through a lot of the Answers and on Splunk's site but can't really find the answer. This group is outstanding, so I'm leaning on you. Any insight is appreciated. pstein ... View more

I have been seeing terrible search time results of late and found Splunk to be running SplunkDeploymentMonitor/bin/scripted_inputs/dm_backfill_factory.py Continuously which in turn chews up resources. When comparing this installation against a "like" install the other install does not run this Python script. What is the reason this might be running? I am looking for a good reason to turn it off without interrupting anything else within Splunk. So far I have come up empty on what is causing it to run. Thoughts? ... View more