How to configure a universal forwarder to add multiple fields to events being forwarded via _meta?

[vinceskahan]

We're trying to find a way to have the universal forwarder send data to the indexer essentially pre-marked with a small number of custom fields (or the like) that we can later search on. For example, a particular computer might be from project-X and be in a environment of test or prod or development. Since VMs come and go, we can't do any persistent mapping of which computer has these added characteristics (host-n.n.n.n might be dev today, prod tomorrow), but the 'data' is persistent.

I stumbled across the _meta construct in inputs.conf, which works well enough for 'one' custom field. Just like specifying which index to use, I also specify _meta = somename::value in inputs.conf.

The question I have is, how could I have 'multiple' such added fields specified by the universal forwarder? I know there is folklore saying doing this on the forwarder side is somehow evil or something, but we're talking about adding under a half-dozen custom fields (?) for all the events coming from the forwarder computer.

Any suggestions other than pointers to the impossibly unreadable/abstract/no-examples docs which I've wasted tens of hours on already?

[vinceskahan · Accepted solution]

Found an answer that seems to work....

_meta = key1::value1 key2::value2
(and so on - use double colons to separate key/value, and whitespace to separate multiple key/value pairs from each other)

tested in inputs.conf on a universalforwarder using current splunk

[somesoni2]

Have a look at this

http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Addfieldsatsearchtime (recommended)
http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Configureindex-timefieldextraction

This gives you to configure adding custom fields. This can be set for a host as well so you can assign custom fields to all data coming from a host/universal forwarder.

[vinceskahan]

Not seeing the relevance. The first link doesn't resolve, and the second one is search-engine-side. The indexer and search engine do not know what categorization we want to apply to the client computers, but the client forwarder systems 'do'. We set custom facts in puppet deploying the forwarder configs that we can use to generate the inputs.conf (etc.) on the forwarder from ERB templates. Again - I am trying to do 'client' side categorization of the data at forwarding time. Is this somehow not possible in splunk ?

[somesoni2]

Well, both the options I suggested manipulate based on client (host value coming from forwarder) only.

However, I can suggest one workaround using which you can send/set multiple values that can be later used for categorization of the data coming from the client. If this doesn't help you get what you want, I hope someone else in the community can provide a better solution.
What you can do is that you can set value of host (this can be set in inputs.conf on UF) to a combination of whatever values you want to assign, say concatenated by colon or something. They will be set on Client (with whatever method you've in place). On indexer side, you extract those categorization fields and put into separate fields which you can use during searching/reporting.

So, on inputs.conf (on UF)

[monitor://foo/bar/something.log]
index=blah
sourcetype=blahblah
host=customPrefix:field1:field2:field3

For index time field extraction, put on indexer, else on Search Head

props.conf
[host:customPrefix*]
REPORT-extractcustom = customfields    ***for search time field extractions
TRANSFORMS-extractcustom = customfields   ***for index time field extraction

transforms.conf
[customfields]
SOURCE_KEY = MetaData:Host
REGEX = customPrefix:(?<Field1>\w+):(?<Field2>\w+):(?<Field3>\w+)