Manually setting source name

[cboillot]

We have discovered that on one of our servers, we had an error in the monitoring stanza and was not getting the logs for several directories.

We can go back and get those logs from the backups. These logs would be restored to a new temp folder, I think it would be something like /datatemp/. How would I set it up to pull this logs in, but without the /datatemp/ in the source ?

[martin_mueller · Accepted solution]

Set up a transforms.conf stanza that removes a leading /datatemp from the source metadata field, and activate it in a props.conf stanza [source::/datatemp/...]. Set this on the first cooking instance, usually indexers but could be heavy forwarders in some deployments.

[cboillot]

Thanks. I totally missed the metadata on the first read through.

For those that end up finding, looking for a solution, here is what I used:

PROPS

[source::...datatemp...]
TRANSFORMS-setsource = correct_source

TRANSFORMS

[correct_source]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = .+log1(.+)
FORMAT = /correct/path$1

Just a few things to note:
path that data was restored to: /datatemp/restore/log1/

WHY/HOW THIS WORKS

Let's go through this step by step.

In props.conf

[source::...datatemp...]

This will apply the following property to anything with datatemp in the path.

TRANSFORMS-setsource = correct_source

This tells Splunk what transforms stanza to apply to the data. In this case, it is applying the correct_source stanza. "setsource" is an identifier to let us admins know what that transform does.

In the transforms.conf

[correct_source]

This is the name we gave this stanza. This will be what is referenced in props.conf.

SOURCE_KEY = MetaData:Source

This is the key that we are changing, the log's source. SOURCE_KEY is the key that we are going to apply the regex to. We are not going to apply this to the data itself, just the log's metadata, and in this cause, the log's source. For more info on what key to use, take a look at the KEY subheading on the transforms.conf page.

DEST_KEY = MetaData:Source

Once the regex is applied, this line tells Splunk where to put that data.

REGEX = .+log1(.+)

This looks for the string 'log1' anywhere in the file path. For excample. the path /datatemp/restore/log/ would not be matched, but /datatemp/restore/log1/ would be. Also, what this regex does, it takes anything after log1, the .+ means any character, and the parentheses groups the results together) and puts in in a group, it 'remembers it' to use later. So out of /datatemp/restore/log1/dir1/file1.log, it would take /dir1/file1.log and make that group 1.

FORMAT = /correct/path$1

Once it has found something with the regex, it passes that to this line. This tells Spunk what we want the end result to look like. So we want to take the group from above, and place it after /correct/path, so in our excample we would get a new source of /correct/path/dir1/file1.log