Getting Data In

Manually setting source name

cboillot
Contributor

We have discovered that on one of our servers, we had an error in the monitoring stanza and was not getting the logs for several directories.

We can go back and get those logs from the backups. These logs would be restored to a new temp folder, I think it would be something like /datatemp/. How would I set it up to pull this logs in, but without the /datatemp/ in the source ?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Set up a transforms.conf stanza that removes a leading /datatemp from the source metadata field, and activate it in a props.conf stanza [source::/datatemp/...]. Set this on the first cooking instance, usually indexers but could be heavy forwarders in some deployments.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Set up a transforms.conf stanza that removes a leading /datatemp from the source metadata field, and activate it in a props.conf stanza [source::/datatemp/...]. Set this on the first cooking instance, usually indexers but could be heavy forwarders in some deployments.

cboillot
Contributor

Thanks. I totally missed the metadata on the first read through.

For those that end up finding, looking for a solution, here is what I used:

PROPS

[source::...datatemp...]
TRANSFORMS-setsource = correct_source

TRANSFORMS

[correct_source]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = .+log1(.+)
FORMAT = /correct/path$1

Just a few things to note:
path that data was restored to: /datatemp/restore/log1/

WHY/HOW THIS WORKS

Let's go through this step by step.

In props.conf

[source::...datatemp...]

This will apply the following property to anything with datatemp in the path.

TRANSFORMS-setsource = correct_source

This tells Splunk what transforms stanza to apply to the data. In this case, it is applying the correct_source stanza. "setsource" is an identifier to let us admins know what that transform does.

In the transforms.conf

[correct_source]

This is the name we gave this stanza. This will be what is referenced in props.conf.

SOURCE_KEY = MetaData:Source

This is the key that we are changing, the log's source. SOURCE_KEY is the key that we are going to apply the regex to. We are not going to apply this to the data itself, just the log's metadata, and in this cause, the log's source. For more info on what key to use, take a look at the KEY subheading on the transforms.conf page.

DEST_KEY = MetaData:Source

Once the regex is applied, this line tells Splunk where to put that data.

REGEX = .+log1(.+)

This looks for the string 'log1' anywhere in the file path. For excample. the path /datatemp/restore/log/ would not be matched, but /datatemp/restore/log1/ would be. Also, what this regex does, it takes anything after log1, the .+ means any character, and the parentheses groups the results together) and puts in in a group, it 'remembers it' to use later. So out of /datatemp/restore/log1/dir1/file1.log, it would take /dir1/file1.log and make that group 1.

FORMAT = /correct/path$1

Once it has found something with the regex, it passes that to this line. This tells Spunk what we want the end result to look like. So we want to take the group from above, and place it after /correct/path, so in our excample we would get a new source of /correct/path/dir1/file1.log

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...