How do I add metadata to events coming from a Splunk forwarder?

[mctester]

We have Splunk running on a server at each of our locations. The NIC on the server has two IPs, one is a unique IP for communications back to our headquarters, the second IP is the same on all servers and is used to communicate with the cash registers running Snare agents to forward their windows events.

The way that the application that manages the terminal works, every location has a Terminal1, Terminal2, and Terminal3. The hostname and IP addresses are the same at every location. So, when a terminal generates a Windows security log message, we can't tell which location it originated at since the hostname/IP are the same everywhere.

How can I add some metadata to the UDP input so that we can uniquely identify each terminal?

[Mick · Accepted solution]

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

[jambajuice]

Is the syntax for the props.conf file correct:

[source::udp:1030]

rather than

[source:udp:1030] or [source::udp::1030]

[jambajuice]

They are heavy forwarders, but we could turn them into light forwarders if this will solve the problem.

We're not using distributed search, just a single central splunk server.

How do I implement this on a light/heavy forwarder?

[jambajuice]

The servers in the stores are forwarding to a central splunk server. They listen on UDP 1030 for incoming syslog messages from snare agents running on the POS terminals. The terminals in each store all use the same names (term1, term2, term3).

The forwarders aren't indexing events. They are just forwarding. So, when we look at the splunk_server field, they all appear as the central splunk server.

The documentation says you can't add metadata to UDP ports. Since we need to push this change out to 500+ stores, I'd like to use the deployment server. If we can add metadata to UDP ports, can we use a variable like:

_meta = $HOSTNAME

Thx.

[gkanapathy]

what type of forwarder? light or heavy?

[gkanapathy · Accepted solution]

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

[jokertothequinn]

This does create the field. However, it doesn't seems to be a metatag, as the field is not working with tstats

for example:

|tstats count where index=main location=* by sourcetype

Following error appears:
When used for 'tstats' searches, the 'WHERE' clause can contain only indexed fields. Ensure all fields in the 'WHERE' clause are indexed. Properly indexed fields should appear in fields.conf.

[scelikok]

Hi @jokertothequinn,

In order to query custom indexed fields you should add them in fields.conf on search heads;

[location]
INDEXED=true

[jokertothequinn]

Oh yeah I did that.
also, I was making use of REPORT instead of TRANSFORM in props.conf

this is what worked:

Props.conf

[source::ping]
TRANSFORMS-add_static_fields = mystaticFieldValue

Transforms.conf

[mystaticFieldValue]
SOURCE_KEY = _raw
WRITE_META = true
REGEX = (.*)
FORMAT = item::31

Fields.conf

[item]
INDEXED = true

[Mick · Accepted solution]

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

[vinceskahan]

works great on 6.1.2 but given the date of the reply I'm wondering if it's deprecated now. Great having one way to do it so thanks !

[Lowell]

This still works in recent version of Splunk. A few versions complained about this as an unknown configuration (not documented in the README), but it should have continued to work throughout various versions.

[vinceskahan]

indeed - I might add (just to document it) that to have multiple metadata tags in your forwarder, just do something like the following space-delimited

_meta=key1::value1 key2::value2 key3::value3

[Genti]

This works like a charm!!!!

[gkanapathy]

I believe he's not using a forwarder, but sending via Snare UDP syslog