Getting Data In

How do I add metadata to events coming from a Splunk forwarder?

mctester
Communicator

We have Splunk running on a server at each of our locations. The NIC on the server has two IPs, one is a unique IP for communications back to our headquarters, the second IP is the same on all servers and is used to communicate with the cash registers running Snare agents to forward their windows events.

The way that the application that manages the terminal works, every location has a Terminal1, Terminal2, and Terminal3. The hostname and IP addresses are the same at every location. So, when a terminal generates a Windows security log message, we can't tell which location it originated at since the hostname/IP are the same everywhere.

How can I add some metadata to the UDP input so that we can uniquely identify each terminal?

2 Solutions

Mick
Splunk Employee
Splunk Employee

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

View solution in original post

jambajuice
Communicator

Is the syntax for the props.conf file correct:

[source::udp:1030]

rather than

[source:udp:1030] or [source::udp::1030]

0 Karma

jambajuice
Communicator

They are heavy forwarders, but we could turn them into light forwarders if this will solve the problem.

We're not using distributed search, just a single central splunk server.

How do I implement this on a light/heavy forwarder?

0 Karma

jambajuice
Communicator

The servers in the stores are forwarding to a central splunk server. They listen on UDP 1030 for incoming syslog messages from snare agents running on the POS terminals. The terminals in each store all use the same names (term1, term2, term3).

The forwarders aren't indexing events. They are just forwarding. So, when we look at the splunk_server field, they all appear as the central splunk server.

The documentation says you can't add metadata to UDP ports. Since we need to push this change out to 500+ stores, I'd like to use the deployment server. If we can add metadata to UDP ports, can we use a variable like:

_meta = $HOSTNAME

Thx.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

what type of forwarder? light or heavy?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

If I am understanding correctly, each Splunk instance gets data that looks the same, and when you use distributed search, the results from each place can't be distinguished? If that's so, all you would need to do is display (or search on) the field splunk_server to distinguish the events at each location.

Update:

So in fact the Splunk instances are forwarders, not indexers. Are they light or heavy? This can be done, but how will be different depending. With a light forwarder, you can just add a key to the input stanza and use a transform on the indexer to write the key to a field. With a heavy forwarder, the transform occurs on the forwarder. You will not not be able to use environment variables in the key value field, unfortunately. This ability is targets for version 4.0.11 however.

inputs:

[udp://1030]
location = mylocation
# in some future release, you should be able to specify location = $HOSTNAME

props:

[source::udp:1030]
TRANSFORMS-location = addlocation

transforms:

[addlocation]
SOURCE_KEY = location
REGEX = (.*)
FORMAT = location::$1
WRITE_META = true

Will add an indexed field location to your data.

jokertothequinn
Path Finder

This does create the field. However, it doesn't seems to be a metatag, as the field is not working with tstats

for example:

|tstats count where index=main location=* by sourcetype

 

Following error appears:
When used for 'tstats' searches, the 'WHERE' clause can contain only indexed fields. Ensure all fields in the 'WHERE' clause are indexed. Properly indexed fields should appear in fields.conf.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @jokertothequinn,

In order to query custom indexed fields you should add them in fields.conf on search heads;

[location]
INDEXED=true

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

jokertothequinn
Path Finder

Oh yeah I did that.
also, I was making use of REPORT instead of TRANSFORM in props.conf

this is what worked:

Props.conf

[source::ping]
TRANSFORMS-add_static_fields = mystaticFieldValue

Transforms.conf

[mystaticFieldValue]
SOURCE_KEY = _raw
WRITE_META = true
REGEX = (.*)
FORMAT = item::31

Fields.conf

[item]
INDEXED = true

0 Karma

Mick
Splunk Employee
Splunk Employee

On the forwarder, you can specify a new metadata field in inputs.conf using the '_meta' setting. For example -

[WinEventLog:Security]
disabled = 0
_meta = Terminal::1

This will added to all events coming in from this input source, and will appear as an indexed field on your indexing instance

vinceskahan
Path Finder

works great on 6.1.2 but given the date of the reply I'm wondering if it's deprecated now. Great having one way to do it so thanks !

Lowell
Super Champion

This still works in recent version of Splunk. A few versions complained about this as an unknown configuration (not documented in the README), but it should have continued to work throughout various versions.

0 Karma

vinceskahan
Path Finder

indeed - I might add (just to document it) that to have multiple metadata tags in your forwarder, just do something like the following space-delimited

_meta=key1::value1 key2::value2 key3::value3
0 Karma

Genti
Splunk Employee
Splunk Employee

This works like a charm!!!!

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I believe he's not using a forwarder, but sending via Snare UDP syslog

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...