https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79679#M2370 <P>Do you have representative sample events?</P> Fri, 28 Jun 2013 19:24:38 GMT martin_mueller 2013-06-28T19:24:38Z https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79678#M2369 <P>Ahoy. We've been experiencing a search performance problem and I'm having trouble figuring out what to do about it. I've been following the advice and techniques outlined here:</P> <P><A href="http://wiki.splunk.com/Community:PerformanceTroubleshooting">http://wiki.splunk.com/Community:PerformanceTroubleshooting</A></P> <P>The very simple search I am performing is this:<BR /> <PRE><BR /> splunk&gt; sourcetype="mysource"<BR /> </PRE><BR /> ...for the last 60 minutes. The search trudges away revealing "0 matching events" for about 10 minutes before revealing anything. During this period, the CPU performing the search shoots to 100% usage. iostat reports low tps (like ~10). vmstat shows no swap activity. So I'm pretty sure this is CPU bound. The search log shows AMPLE time consumption by "SearchOperator:kv", and all of the default kv searches are being performed. The "access-extractions" system default extraction seems to take about 5 whole minutes. The job inspector reports practically all of the time in "dispatch.evaluate.search". (I found it odd that the job inspector didn't identify command.search.kv considering the contents of the search log.) Note that according to the high-level summary on the search page, "mysource" has about 276 million events. Not sure if this is a lot or not. Any ideas?</P> <P>UPDATE:</P> <P>So many of the time-consuming "SearchOperator:kv" lines in the search.log file seem to be coming from specs in the config files that are not related to my custom sourcetype. For example, the access-extractions transform is referenced by a bunch of default sourcetype specs, but not by my custom sourcetype spec. The following search:<BR /> <PRE><BR /> | metadata type=sourcetypes index="main"<BR /> </PRE><BR /> ...shows a single result that is my custom sourcetype.</P> Fri, 28 Jun 2013 18:54:02 GMT https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79678#M2369 bcavagnolo 2013-06-28T18:54:02Z https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79679#M2370 <P>Do you have representative sample events?</P> Fri, 28 Jun 2013 19:24:38 GMT https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79679#M2370 martin_mueller 2013-06-28T19:24:38Z https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79680#M2371 <P>Yes. Practically all of the events in splunk are of of the sourcetype that I specify. And the results do eventually appear. It just takes 10 minutes.</P> Fri, 28 Jun 2013 22:20:36 GMT https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79680#M2371 bcavagnolo 2013-06-28T22:20:36Z https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79681#M2372 <P>For the record, I think we figured out the root cause of this issue. We ended up having a crazy number of sourcetypes defined. We ended up rebuilding the index on a fresh splunk install. All is well now.</P> Fri, 23 Aug 2013 23:05:53 GMT https://community.splunk.com/t5/Splunk-ITSI/CPU-bound-search-seems-related-to-KV-processing/m-p/79681#M2372 bcavagnolo 2013-08-23T23:05:53Z