Getting Data In

Copy the value of a metadata filed to a new field at Index time

MedralaG
Communicator

I have some network devices sending logs to a syslog server that has a UF installed. The 'host' field is populated with the name of the UF host.
We want the 'host' to be overwritten with the name of the hosts that are generating the logs initially (easy) however we also want to create an additional field ('forwarder') which will keep the name of the UF host.

I believe the best way of doing this would be to use transforms to first copy the value of the 'host' field to the new 'forwarder' field however It doesn't seem to be working with the following stanza

props.conf
[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf
[copy_host_name]
SOURCE_KEY = MetaData::host
REGEX = (.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)
[forwarder]
INDEXED=true

any suggestions?

0 Karma
1 Solution

MedralaG
Communicator

Just for the record, I managed to find the problem that was causing my stanza in the transforms.conf not to work.
1. I had to remove the additional : in the SOURCEKEY stanza. This will allow me to pull the data from the HOST metadata field into the transforms.
2. The value pulled from the SOURCEKEY = MetaData:host will appear in the following format:

host::*hostname*

This means that I have to adjust the REGEX to exclude the host::

props.conf

[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf

[copy_host_name]
SOURCE_KEY = MetaData:host
REGEX = host::(.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)

[forwarder]
INDEXED=true

With these corrections made I am successfully able to copy the value of the HOST field to a newly created field (at index time) and then I'm free overwrite the original host field.

View solution in original post

MedralaG
Communicator

Just for the record, I managed to find the problem that was causing my stanza in the transforms.conf not to work.
1. I had to remove the additional : in the SOURCEKEY stanza. This will allow me to pull the data from the HOST metadata field into the transforms.
2. The value pulled from the SOURCEKEY = MetaData:host will appear in the following format:

host::*hostname*

This means that I have to adjust the REGEX to exclude the host::

props.conf

[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf

[copy_host_name]
SOURCE_KEY = MetaData:host
REGEX = host::(.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)

[forwarder]
INDEXED=true

With these corrections made I am successfully able to copy the value of the HOST field to a newly created field (at index time) and then I'm free overwrite the original host field.

woodcock
Esteemed Legend

Good job on updating us! Now close it all off by clicking Accept on your answer.

0 Karma

MedralaG
Communicator

Done, thank you.

woodcock
Esteemed Legend

First, do your syslog this way:
http://www.georgestarcher.com/splunk-success-with-syslog/

But do it with 1 wrinkle: make sure that your syslog host name is also in the file path.
Then you can get at it any time that you like with this:

SOURCE_KEY = MetaData::source

Or something like this:

| rex field=source "(?:[^\/]+\/){4}(?<syslog_host>[^\/]+)" 
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi MedralaG,
I'd use a different approach:
I'd replace host with the correct host value (as you're doing) and I'd create a lookup with two columns:

  • host of appliance sending logs to a UF,
  • UF name.

In this way, you always have both the information, host and related UF, in an easier way.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...