topic How do I add metadata to events coming from a Splunk forwarder? in Getting Data In

How do I add metadata to events coming from a Splunk forwarder?

mctester — Fri, 16 Apr 2010 04:31:01 GMT

We have Splunk running on a server at each of our locations. The NIC on the server has two IPs, one is a unique IP for communications back to our headquarters, the second IP is the same on all servers and is used to communicate with the cash registers running Snare agents to forward their windows events.

The way that the application that manages the terminal works, every location has a Terminal1, Terminal2, and Terminal3. The hostname and IP addresses are the same at every location. So, when a terminal generates a Windows security log message, we can't tell which location it originated at since the hostname/IP are the same everywhere.

How can I add some metadata to the UDP input so that we can uniquely identify each terminal?

Re: How do I add metadata to events coming from a Splunk forwarder?

Mick — Fri, 16 Apr 2010 04:33:40 GMT

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

Re: How do I add metadata to events coming from a Splunk forwarder?

gkanapathy — Fri, 16 Apr 2010 06:34:51 GMT

I believe he's not using a forwarder, but sending via Snare UDP syslog

Re: How do I add metadata to events coming from a Splunk forwarder?

gkanapathy — Fri, 16 Apr 2010 06:49:51 GMT

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

Re: How do I add metadata to events coming from a Splunk forwarder?

jambajuice — Fri, 16 Apr 2010 07:01:14 GMT

The servers in the stores are forwarding to a central splunk server. They listen on UDP 1030 for incoming syslog messages from snare agents running on the POS terminals. The terminals in each store all use the same names (term1, term2, term3).

The forwarders aren't indexing events. They are just forwarding. So, when we look at the splunk_server field, they all appear as the central splunk server.

The documentation says you can't add metadata to UDP ports. Since we need to push this change out to 500+ stores, I'd like to use the deployment server. If we can add metadata to UDP ports, can we use a variable like:

_meta = $HOSTNAME

Thx.

Re: How do I add metadata to events coming from a Splunk forwarder?

gkanapathy — Fri, 16 Apr 2010 15:31:32 GMT

what type of forwarder? light or heavy?

Re: How do I add metadata to events coming from a Splunk forwarder?

jambajuice — Sat, 17 Apr 2010 00:57:47 GMT

They are heavy forwarders, but we could turn them into light forwarders if this will solve the problem.

We're not using distributed search, just a single central splunk server.

How do I implement this on a light/heavy forwarder?

Re: How do I add metadata to events coming from a Splunk forwarder?

jambajuice — Tue, 11 May 2010 01:38:39 GMT

Is the syntax for the props.conf file correct:

[source::udp:1030]

rather than

[source:udp:1030] or [source::udp::1030]

Re: How do I add metadata to events coming from a Splunk forwarder?

Genti — Fri, 17 Jun 2011 00:53:00 GMT

This works like a charm!!!!

Re: How do I add metadata to events coming from a Splunk forwarder?

vinceskahan — Wed, 30 Dec 2015 19:51:59 GMT

works great on 6.1.2 but given the date of the reply I'm wondering if it's deprecated now. Great having one way to do it so thanks !

Re: How do I add metadata to events coming from a Splunk forwarder?

Lowell — Wed, 01 Mar 2017 23:02:12 GMT

This still works in recent version of Splunk. A few versions complained about this as an unknown configuration (not documented in the README), but it should have continued to work throughout various versions.

Re: How do I add metadata to events coming from a Splunk forwarder?

vinceskahan — Wed, 01 Mar 2017 23:57:57 GMT

indeed - I might add (just to document it) that to have multiple metadata tags in your forwarder, just do something like the following space-delimited

_meta=key1::value1 key2::value2 key3::value3

Re: How do I add metadata to events coming from a Splunk forwarder?

jokertothequinn — Mon, 05 Feb 2024 10:05:02 GMT

This does create the field. However, it doesn't seems to be a metatag, as the field is not working with tstats

for example:

|tstats count where index=main location=* by sourcetype

Following error appears:
When used for 'tstats' searches, the 'WHERE' clause can contain only indexed fields. Ensure all fields in the 'WHERE' clause are indexed. Properly indexed fields should appear in fields.conf.

Re: How do I add metadata to events coming from a Splunk forwarder?

scelikok — Mon, 05 Feb 2024 10:24:16 GMT

Hi @jokertothequinn,

In order to query custom indexed fields you should add them in fields.conf on search heads;

[location] INDEXED=true

Re: How do I add metadata to events coming from a Splunk forwarder?

jokertothequinn — Mon, 05 Feb 2024 10:46:49 GMT

Oh yeah I did that.
also, I was making use of REPORT instead of TRANSFORM in props.conf

this is what worked:

Props.conf

[source::ping]
TRANSFORMS-add_static_fields = mystaticFieldValue

Transforms.conf

[mystaticFieldValue]
SOURCE_KEY = _raw
WRITE_META = true
REGEX = (.*)
FORMAT = item::31

Fields.conf

[item]
INDEXED = true