Getting Data In

How do I add metadata to events coming from a Splunk forwarder?

Communicator

We have Splunk running on a server at each of our locations. The NIC on the server has two IPs, one is a unique IP for communications back to our headquarters, the second IP is the same on all servers and is used to communicate with the cash registers running Snare agents to forward their windows events.

The way that the application that manages the terminal works, every location has a Terminal1, Terminal2, and Terminal3. The hostname and IP addresses are the same at every location. So, when a terminal generates a Windows security log message, we can't tell which location it originated at since the hostname/IP are the same everywhere.

How can I add some metadata to the UDP input so that we can uniquely identify each terminal?

2 Solutions

Splunk Employee
Splunk Employee

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

View solution in original post

Splunk Employee
Splunk Employee

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

View solution in original post

Communicator

Is the syntax for the props.conf file correct:

[source::udp:1030]

rather than

[source:udp:1030] or [source::udp::1030]

0 Karma

Communicator

They are heavy forwarders, but we could turn them into light forwarders if this will solve the problem.

We're not using distributed search, just a single central splunk server.

How do I implement this on a light/heavy forwarder?

0 Karma

Communicator

The servers in the stores are forwarding to a central splunk server. They listen on UDP 1030 for incoming syslog messages from snare agents running on the POS terminals. The terminals in each store all use the same names (term1, term2, term3).

The forwarders aren't indexing events. They are just forwarding. So, when we look at the splunk_server field, they all appear as the central splunk server.

The documentation says you can't add metadata to UDP ports. Since we need to push this change out to 500+ stores, I'd like to use the deployment server. If we can add metadata to UDP ports, can we use a variable like:

_meta = $HOSTNAME

Thx.

0 Karma

Splunk Employee
Splunk Employee

what type of forwarder? light or heavy?

0 Karma

Splunk Employee
Splunk Employee

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

View solution in original post

Splunk Employee
Splunk Employee

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

View solution in original post

Path Finder

works great on 6.1.2 but given the date of the reply I'm wondering if it's deprecated now. Great having one way to do it so thanks !

Super Champion

This still works in recent version of Splunk. A few versions complained about this as an unknown configuration (not documented in the README), but it should have continued to work throughout various versions.

0 Karma

Path Finder

indeed - I might add (just to document it) that to have multiple metadata tags in your forwarder, just do something like the following space-delimited

_meta=key1::value1 key2::value2 key3::value3
0 Karma

Splunk Employee
Splunk Employee

This works like a charm!!!!

0 Karma

Splunk Employee
Splunk Employee

I believe he's not using a forwarder, but sending via Snare UDP syslog

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!