topic Re: Copy the value of a metadata filed to a new field at Index time in Getting Data In

Copy the value of a metadata filed to a new field at Index time

MedralaG — Tue, 29 Sep 2020 14:49:00 GMT

I have some network devices sending logs to a syslog server that has a UF installed. The 'host' field is populated with the name of the UF host.
We want the 'host' to be overwritten with the name of the hosts that are generating the logs initially (easy) however we also want to create an additional field ('forwarder') which will keep the name of the UF host.

I believe the best way of doing this would be to use transforms to first copy the value of the 'host' field to the new 'forwarder' field however It doesn't seem to be working with the following stanza

props.conf
[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf
[copy_host_name]
SOURCE_KEY = MetaData::host
REGEX = (.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)
[forwarder]
INDEXED=true

any suggestions?

Re: Copy the value of a metadata filed to a new field at Index time

gcusello — Sat, 08 Jul 2017 06:46:21 GMT

Hi MedralaG,
I'd use a different approach:
I'd replace host with the correct host value (as you're doing) and I'd create a lookup with two columns:

host of appliance sending logs to a UF,
UF name.

In this way, you always have both the information, host and related UF, in an easier way.
Bye.
Giuseppe

Re: Copy the value of a metadata filed to a new field at Index time

woodcock — Sat, 08 Jul 2017 14:39:52 GMT

First, do your syslog this way:
http://www.georgestarcher.com/splunk-success-with-syslog/

But do it with 1 wrinkle: make sure that your syslog host name is also in the file path.
Then you can get at it any time that you like with this:

SOURCE_KEY = MetaData::source

Or something like this:

| rex field=source "(?:[^\/]+\/){4}(?<syslog_host>[^\/]+)"

Re: Copy the value of a metadata filed to a new field at Index time

MedralaG — Wed, 09 Aug 2017 10:58:30 GMT

Just for the record, I managed to find the problem that was causing my stanza in the transforms.conf not to work.
1. I had to remove the additional : in the SOURCEKEY stanza. This will allow me to pull the data from the HOST metadata field into the transforms.
2. The value pulled from the SOURCEKEY = MetaData:host will appear in the following format:

host::*hostname*

This means that I have to adjust the REGEX to exclude the host::

props.conf

[source::/var/log/messages]
TRANSFORMS-t1=copy_host_name

transforms.conf

[copy_host_name]
SOURCE_KEY = MetaData:host
REGEX = host::(.*)
FORMAT = forwarder::$1
WRITE_META = true

fields.conf (on the search head)

[forwarder]
INDEXED=true

With these corrections made I am successfully able to copy the value of the HOST field to a newly created field (at index time) and then I'm free overwrite the original host field.

Re: Copy the value of a metadata filed to a new field at Index time

woodcock — Wed, 09 Aug 2017 11:19:53 GMT

Good job on updating us! Now close it all off by clicking Accept on your answer.

Re: Copy the value of a metadata filed to a new field at Index time

MedralaG — Wed, 09 Aug 2017 11:37:05 GMT

Done, thank you.