command.search.kv performance

[rettops]

We have a search that is spending most of its time in command.search.kv. If we give it a search which doesn't need any fields, like:

index=myindex sourcetype=mytype | stats count

it takes 1.8 seconds to count the 104,000 events (that alone seems high).

If instead we give it any search which would cause it to extract fields, e.g.:

index=myindex sourcetype=mytype myfield=val1 | stats count

then the time jumps up to 9 seconds, almost all of which is in command.search.kv.

We have no custom field extractions, field transformations, field aliases or tags. Each data item does have 101 fields, but the search in question really only needs 8 of them. Is there any way to speed things up? For example, is there a way to tell Splunk to turn off all of the automatic field extractions and have it use only some user defined ones? Alternatively, is there any way to debug what's happening during command.search.kv?

[martin_mueller · Accepted solution]

You can set KV_MODE to none in props.conf to turn off automatic extraction of key-value pairs.

[landen99]

Extractions only happen for fields needed by the search.

[rettops]

I forgot to mention - the 9 seconds is in 'fast' mode search. In verbose mode it jumps to 18 seconds.