command.search.kv in search.log

[jariw]

Hi,

Difficult question...

Whe have some problems with search performance. Looking at the job inspector i noticed within the slow jobs the command.search.kv is taking a lot of time. 

What is this? And where is this part of the search-command executed (indexer or search-head)?

I notice especialy wineventlogs are taking a lot of this kv time.

I created a blank SH, no apps at all, timed some searches with some different indexes and installed some different apps. I noticed when this command.search.kv takes more time. Sometimes this is correct in relation to the app/event match if looking at the props.conf.  Turning the right app  off makes this command.search.kv decrease a lot to almost zero.

But with winevents.. no go.. it stays high.  Also even without the fieldextracts etc installed on this blank SH, most fields are extracted. If those field were extracted at index time.. i can imagine there wil be no command.search.kv time wasted (wild guess). does the indexer extract these fields at search time (strange strange) and wil this be the command.search.kv??

So is it possible this command.search.kv also run's on the indexers? And so.. does this lookup / field extraction cost most off the time?

Thanks in advance

greets Jari

[jariw]

A search with specific  start/end date (for the test always the same) took 223.179 secs.. from which 117.82 secs.  It always seems the time with this command.search.kv is almost half the time of search duration.

We have a clustered site. 12 indexers and smartstore. First bvlame was smartstore.. but the buckets are in cache so no smartstore anymore.

The question is why are the fields extracted (from wineventlogs) when there are no fieldextractions in the apps (or are winevents extracted buy default because the server is "vanilla"). It seems to cost a lot off time to extract those.

[livehybrid]

Hi @jariw 

How much time is the command.search.kv step taking? This is the field extraction phase and covers things like regex extractions which can be resource intensive. 

What does you current setup look like? Do you have custom regex field extractions or purely from Splunkbase apps?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing