Splunk Search
Highlighted

How to configure a universal forwarder to add search-time metadata to all events?

New Member

Hi Everyone,

Our setup is a universal forwarder --> heavy forwarder --> indexer. I am looking to modify a universal forwarder config so I can search on static metadata in Splunk Web. For example, I'd like to be able to search for an app_name, build_version, or environment_name that would be set when the instance comes up.

I have seen various posts on this site about accomplishing that and most of them come back to the link below. This seems like the correct path, but many of the keys are out of date. I have finally settled on the structure below for my files, but I am not seeing anything in Splunk Web. Is this outcome just not possible with Splunk, or am I missing something?

props.conf:

[host::i-e420f63c]
TRANSFORMS-test = MYTRANSFORM

transforms.conf:

[MYTRANSFORM]
REGEX = .*?
SOURCE_KEY = _raw
FORMAT = instance::app_name

https://answers.splunk.com/answers/39405/adding-static-field-value-using-props-transforms-based-on-s...

0 Karma
Highlighted

Re: How to configure a universal forwarder to add search-time metadata to all events?

SplunkTrust
SplunkTrust

This transform should happen on Heavy forwarder and check the REGEX and FORMAT (not matching with the post https://answers.splunk.com/answers/39405/adding-static-field-value-using-props-transforms-based-on-s...)

0 Karma
Highlighted

Re: How to configure a universal forwarder to add search-time metadata to all events?

New Member

@somesoni2 what should the REGEX and FORMAT be? I have tried the syntax from the previous post and that doesn't work either. These values would be set at instance creation time and are not extracting from anything else.

Also, why is the heavy forwarder the only option? We have hundreds of instances that come and go so adding this to a HF doesn't sound like a feasible option.

0 Karma
Highlighted

Re: How to configure a universal forwarder to add search-time metadata to all events?

SplunkTrust
SplunkTrust

I can suggest one workaround using which you can send/set multiple values that can be later used for categorization of the data coming from the forwarders. What you can do is that you can set value of host (this can be set in inputs.conf on UF) to a combination of whatever values you want to assign, say concatenated by colon or something. They will be set on Client .On indexer side, you extract those categorization fields and put into separate fields which you can use during searching/reporting.

So, on inputs.conf (on UF)

 [monitor://foo/bar/something.log]
 index=blah
 sourcetype=blahblah
 host=customPrefix:field1:field2:field3

For index time field extraction, put on HeavyForwarder/indexer, else on Search Head

 props.conf
 [host:customPrefix*]
 REPORT-extractcustom = customfields    ***User this for search time field extractions
 TRANSFORMS-extractcustom = customfields   ***Use this for index time field extraction



 transforms.conf
 [customfields]
 SOURCE_KEY = MetaData:Host
 REGEX = customPrefix:(?<Field1>\w+):(?<Field2>\w+):(?<Field3>\w+)

Update

Well here, you are passing a pre-defined list of fields (just values but in specific order) with a custom prefix so that all hosts following that pattern can be identified during extraction even though they send data for different sourcetypes) from the forwarder using inputs.conf entry. On Search Head (since you just want search time extraction, I would use that term only), the transform is applied for all host with custom prefix and extract the field, again in specific order in which they were sent. E.g. if you want to send appname, buildnumber and environment_name, you'll pass host as trythis:myapp:12345:production. On your transforms, you'll extract them using REGEX = trythis:(?.*):(?.*):(?.*).

Again, the REGEX offers so many things that you can pass dynamic key value pairs as well. Like this

inputs.conf (on UF) host=trythis:#app_name=myapp#:#build_number=12345#:#environment_name=production#
transforms.conf (on SH) REGEX = #(?<_KEY_1>[^=]+)=(?<_VAL_1>[^#]+)

0 Karma
Highlighted

Re: How to configure a universal forwarder to add search-time metadata to all events?

New Member

So I don't want to do anything that will negatively impact index performance. Search-time seems like the best option but how is this different than just searching for those values in the host name? What exactly is the transforms.conf doing in your example? Ideally, I am passing name/value pairs (e.g. app_name=myapp). Does your structure offer anything more?

0 Karma
Highlighted

Re: How to configure a universal forwarder to add search-time metadata to all events?

SplunkTrust
SplunkTrust

Well here, you are passing a pre-defined list of fields (just values but in specific order) with a custom prefix so that all hosts following that pattern can be identified during extraction even though they send data for different sourcetypes) from the forwarder using inputs.conf entry. On Search Head (since you just want search time extraction, I would use that term only), the transform is applied for all host with custom prefix and extract the field, again in specific order in which they were sent. E.g. if you want to send appname, buildnumber and environment_name, you'll pass host as trythis:myapp:12345:production. On your transforms, you'll extract them using REGEX = trythis:(?.*):(?.*):(?.*).

Again, the REGEX offers so many things that you can pass dynamic key value pairs as well. Like this

inputs.conf (on UF) host=trythis:#app_name=myapp#:#build_number=12345#:#environment_name=production#
transforms.conf (on SH) REGEX = #(?<_KEY_1>[^=]+)=(?<_VAL_1>[^#]+)

0 Karma