transforms.conf won't let me change the sourcetype

[llacoste]

Hi all,

I've got a small problem here..

Here is the scenario:

I am receiving a file from a UF which has got this format: /var/log/test.2017-11-26_11_16 (let's say).
My goal is to strip it from the date.

That.. I managed by doing so:

In props.conf:
[test]
TRANSFORMS-changesource = changesource
TRANSFORMS=changesourcetype = changesourcetype

In transforms.conf:
[changesource]
SOURCE_KEY=MetaData:Source
DEST_KEY=MetaData:Source
REGEX=(.*?).\w{4}-

trying to match a bit more on the regex for the non catching group so I am sure I'm not stripping something by accidenteven if it's not necessary I think.

FORMAT = source::$1

if I'm writing source::$1 I see my source having: source::/var/log/ instead of only /var/log/ so I'm thinking about only putting $1 without the leading source:: just tell me if that could be a problem in the future?

So, this far it is working.

However I also want to change the sourcetype to something else. At the moment it is "test" but I would like to change it to "access_combined" for example and I can't change it before for other reasons ( I don't want all the access_combined on the Indexer to strip the date for example).

I am trying this but it doesn't seem to change the sourcetype:

[changesourcetype]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY=MetaData:Sourcetype
FORMAT = sourcetype::access_combined

This should work to my understanding... and I don't need any REGEX field as I am not trying to take this from event or source file name or host or anything else... I just want to give the sourcetype another name and that's all.

Maybe I am not thinking of anything else at the moment and there is an easier solution to this... but I admit I have been trying several things in that stanza and nothing seems to be working... I even tried to remove the SOURCE_KEY, add the REGEX =(.*?) to see if that was the problem... but it seems it never changes this sourcetype..

Thanks for your help, I know it may be something trivial.

[llacoste · Accepted solution]

Actually I solved my own problem 😛

This is what I came up with and it's now working:

into transforms.conf I needed to have this:

  DEST_KEY = MetaData:Sourcetype
    REGEX = (.*?)
    FORMAT = sourcetype::access_combined

The key here is that I was missing the REGEX and nothing was happening because of that... I read the docs it's mandatory:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf
"* REGEX is required for all index-time transforms."

I figured out I needed to match anything/everything and It is now working fine.

Thanks for helping me @hardikJsheth

Cheers

[DUThibault]

Just a note: Not all stanzas in transforms.conf will require REGEX, as extractions can end up listed in that file too (confusing, isn't it?).

Also, your REGEX does not need to capture anything since you're not using the capture result. REGEX = .* will do just fine.

[hardikJsheth]

There is one typo which I assume is there in the question only (= instead of -)

[test]
TRANSFORMS-changesource = changesource
TRANSFORMS-changesourcetype = changesourcetype

For transfroms you don't need SOURCE_KEY.

Can you try with following configuration on your indexer?

[changesourcetype]
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::access_combined

[llacoste]

Hi,

Thanks for your answer.

You're right, the transforms with = was typo on the question only, it's the good one on the actual file. Sorry for that.

Here is my new config:

Props.conf

[test]
TRANSFORMS-changesourcetype = changesourcetype

Transforms.conf

   [changesourcetype]
    DEST_KEY = MetaData:Sourcetype
    FORMAT = sourcetype::access_combined

I actually did a copy paste here, so if there is a typo error it would be an actual error in the config.

It didn't seem to work, I've got this:

_time                           host                source                                                  sourcetype
11/27/17
12:54:44.000 PM 192.168.56.1 /var/log/test.2017-11-26-00_00_00 test

Any idea?

[llacoste]

Adding more info on the tests I just did:

If I use this:

SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = MetaData:Sourcetype
REGEX= (.*?)
FORMAT= sourcetype::$1

I actually don't catch anything and my sourcetype is empty now on the search.

Meaning there is nothing in the SOURCE_KEY?? I was expecting to have "test" ...

Also, If I'm trying to take from the event itself the first word, I can assign the sourcetype to the first word of the event:
DEST_KEY = MetaData:Sourcetype
REGEX = (.*?)\s
FORMAT = sourcetype::$1

The event itself being : "access_combined this is a test"

Now this way it is working and I have sourcetype=access_combined.

Strange.. any idea about what's going wrong here?

Thanks for your help guys.