Getting Data In

Copy the value of a metadata filed to a new field at Index time

Communicator

I have some network devices sending logs to a syslog server that has a UF installed. The 'host' field is populated with the name of the UF host.
We want the 'host' to be overwritten with the name of the hosts that are generating the logs initially (easy) however we also want to create an additional field ('forwarder') which will keep the name of the UF host.

I believe the best way of doing this would be to use transforms to first copy the value of the 'host' field to the new 'forwarder' field however It doesn't seem to be working with the following stanza

props.conf
[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf
[copy_host_name]
SOURCE_KEY = MetaData::host
REGEX = (.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)
[forwarder]
INDEXED=true

any suggestions?

0 Karma
1 Solution

Communicator

Just for the record, I managed to find the problem that was causing my stanza in the transforms.conf not to work.
1. I had to remove the additional : in the SOURCEKEY stanza. This will allow me to pull the data from the HOST metadata field into the transforms.
2. The value pulled from the SOURCEKEY = MetaData:host will appear in the following format:

host::*hostname*

This means that I have to adjust the REGEX to exclude the host::

props.conf

[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf

[copy_host_name]
SOURCE_KEY = MetaData:host
REGEX = host::(.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)

[forwarder]
INDEXED=true

With these corrections made I am successfully able to copy the value of the HOST field to a newly created field (at index time) and then I'm free overwrite the original host field.

View solution in original post

Communicator

Just for the record, I managed to find the problem that was causing my stanza in the transforms.conf not to work.
1. I had to remove the additional : in the SOURCEKEY stanza. This will allow me to pull the data from the HOST metadata field into the transforms.
2. The value pulled from the SOURCEKEY = MetaData:host will appear in the following format:

host::*hostname*

This means that I have to adjust the REGEX to exclude the host::

props.conf

[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf

[copy_host_name]
SOURCE_KEY = MetaData:host
REGEX = host::(.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)

[forwarder]
INDEXED=true

With these corrections made I am successfully able to copy the value of the HOST field to a newly created field (at index time) and then I'm free overwrite the original host field.

View solution in original post

Esteemed Legend

Good job on updating us! Now close it all off by clicking Accept on your answer.

0 Karma

Communicator

Done, thank you.

Esteemed Legend

First, do your syslog this way:
http://www.georgestarcher.com/splunk-success-with-syslog/

But do it with 1 wrinkle: make sure that your syslog host name is also in the file path.
Then you can get at it any time that you like with this:

SOURCE_KEY = MetaData::source

Or something like this:

| rex field=source "(?:[^\/]+\/){4}(?<syslog_host>[^\/]+)" 
0 Karma

Legend

Hi MedralaG,
I'd use a different approach:
I'd replace host with the correct host value (as you're doing) and I'd create a lookup with two columns:

  • host of appliance sending logs to a UF,
  • UF name.

In this way, you always have both the information, host and related UF, in an easier way.
Bye.
Giuseppe

0 Karma