CPU-bound search. seems related to KV processing.

[bcavagnolo]

Ahoy. We've been experiencing a search performance problem and I'm having trouble figuring out what to do about it. I've been following the advice and techniques outlined here:

http://wiki.splunk.com/Community:PerformanceTroubleshooting

The very simple search I am performing is this:

splunk> sourcetype="mysource"

...for the last 60 minutes. The search trudges away revealing "0 matching events" for about 10 minutes before revealing anything. During this period, the CPU performing the search shoots to 100% usage. iostat reports low tps (like ~10). vmstat shows no swap activity. So I'm pretty sure this is CPU bound. The search log shows AMPLE time consumption by "SearchOperator:kv", and all of the default kv searches are being performed. The "access-extractions" system default extraction seems to take about 5 whole minutes. The job inspector reports practically all of the time in "dispatch.evaluate.search". (I found it odd that the job inspector didn't identify command.search.kv considering the contents of the search log.) Note that according to the high-level summary on the search page, "mysource" has about 276 million events. Not sure if this is a lot or not. Any ideas?

UPDATE:

So many of the time-consuming "SearchOperator:kv" lines in the search.log file seem to be coming from specs in the config files that are not related to my custom sourcetype. For example, the access-extractions transform is referenced by a bunch of default sourcetype specs, but not by my custom sourcetype spec. The following search:

| metadata type=sourcetypes index="main"

...shows a single result that is my custom sourcetype.

[bcavagnolo · Accepted solution]

For the record, I think we figured out the root cause of this issue. We ended up having a crazy number of sourcetypes defined. We ended up rebuilding the index on a fresh splunk install. All is well now.

[bcavagnolo]

Yes. Practically all of the events in splunk are of of the sourcetype that I specify. And the results do eventually appear. It just takes 10 minutes.

[martin_mueller]

Do you have representative sample events?