topic Re: What are some of the best practices for field extractions? in Splunk Search

What are some of the best practices for field extractions?

a212830 — Mon, 20 Aug 2018 15:38:13 GMT

Hi,

There is some debate in our group regarding best practices for field extractions. We have a feed that has well defined key-value fields. We also have field extractions setup on the SH, for a number of these fields. Is there a really a need for the field extractions, since key-value pairs will get picked up automatically? Pros/cons? We use CIM/ES extensively.

Re: What are some of the best practices for field extractions?

sloshburch — Mon, 20 Aug 2018 16:00:35 GMT

It depends on the sourcetype definition. If the sourcetype is handling the extractions natively then you are slowing things down by adding more search time extractions.

Sounds like we might need an example...

Re: What are some of the best practices for field extractions?

a212830 — Mon, 20 Aug 2018 16:05:36 GMT

It's pretty straight-forward:

field1=value1 field2=value2 field3=value3

The sourcetype is configured with KVMODE=auto. We also have an app on the search-head, which also does extractions against this sourcetype, using transforms. IMO, the app isn't needed, unless there's some need for it with CIM/ES, which I'm really just getting familiar with.

Re: What are some of the best practices for field extractions?

sloshburch — Wed, 22 Aug 2018 13:30:10 GMT

Yea...def straightforward. My guess is your hunch: a case of over engineering. While they may have considered it benign, it def would produce redundant processing and marginally slow down the Search Head processing.

If I were you, here's what I would do to validate:

Export a sample of the data to my local sandbox and index it there showing that the name/value pairs are available out of the box
Comment out the related reference to the transforms that exists in props (but not the transforms itself as it might be used by other sourcetypes) and see if it still works
Change KV_MODE from auto to auto_escaped. See props.conf.spec

To be safe, you might as well share what the transforms is. You mentioned KV_MODE which is props. But let's be sure about what the purpose of the related transforms is.

Also, it could be the case that in your environment, someone erroneously edited the default KV_MODE thereby obligating any sourcetype to need such over-the-top extra config.