Ahoy. We've been experiencing a search performance problem and I'm having trouble figuring out what to do about it. I've been following the advice and techniques outlined here:
http://wiki.splunk.com/Community:PerformanceTroubleshooting
The very simple search I am performing is this:
splunk> sourcetype="mysource"
UPDATE:
So many of the time-consuming "SearchOperator:kv" lines in the search.log file seem to be coming from specs in the config files that are not related to my custom sourcetype. For example, the access-extractions transform is referenced by a bunch of default sourcetype specs, but not by my custom sourcetype spec. The following search:
| metadata type=sourcetypes index="main"
For the record, I think we figured out the root cause of this issue. We ended up having a crazy number of sourcetypes defined. We ended up rebuilding the index on a fresh splunk install. All is well now.
For the record, I think we figured out the root cause of this issue. We ended up having a crazy number of sourcetypes defined. We ended up rebuilding the index on a fresh splunk install. All is well now.
Yes. Practically all of the events in splunk are of of the sourcetype that I specify. And the results do eventually appear. It just takes 10 minutes.
Do you have representative sample events?