Index time fields with heavy forwarder

sonicZ — Thu, 26 Sep 2013 00:03:30 GMT

Hello,

I have a similar question to
This question

I also know indexed fields are generally a nono but we are going to use one anyway 🙂
I am in a distributed environment with a search head pool AND a heavy forwarder handling certain props and transforms

Would the props.conf, transforms.conf, and fields.conf changes all be made on all of the indexers since the field is created at index time or would it possibly need to be made on the heavy forwarder since it handles additional parsing?

here's what i have
Thanks

props.conf
[logger_cef]
TRANSFORMS-flexString2 = flexString2_indx

transforms.conf
[flexString2_indx]
REGEX = flexString2=(?<flex2>[^ ]+)
FORMAT = flexstring2::"$1"
WRITE_META = true

fields.conf
[flexstring2]
INDEXED=true

Re: Index time fields with heavy forwarder

Lucas_K — Mon, 28 Sep 2020 14:50:42 GMT

Its all about your input data path.

If all your the data is coming from the heavy and then going to the indexer will mean the data is cooked when it hits the indexer. Cooked data is to the immune to then those particular props and transform write_meta field stanza's and will have no effect on the incoming data for those particular sources. You can force the data back through the parsing queues ( more info here : http://answers.splunk.com/answers/97918/reparsing-cooked-data-coming-from-a-heavy-forwarder-possible ) to make it perform those write_meta commands but this has its own issues.

In a nutshell, where ever your data is actually being parsed is where those props and transforms need to live. That being said, having those props and transforms on the indexer won't hurt and may catch sources that directly talk to your indexers (ie. other UF's). This can make management of your configuration harder as you will have several places you need to update config on or check when something doesn't work quite right.

Also in regards to indexed fields you need to be careful as they can cause rapid bucket rolls (I just recently implemented some index time field extractions to improve performance) which will can also have a negative search performance hit for anything accessing the destination indexes ( http://answers.splunk.com/answers/102682/safe-custom-setting-for-maxmetaentries-in-indexesconf ).

topic Re: Index time fields with heavy forwarder in Getting Data In

Index time fields with heavy forwarder

Re: Index time fields with heavy forwarder