Change sourcetype via field extraction and transforms

[niboucher]

Hi there,

One of UF is configured to send logs to sourcetype testData.
I'd like to push some of those logs matching a certain pattern (all logs matching the "[A][B]" pattern) to sourcetype testData_B.

Sample of log

[A][B] blabla
[A][C] blabla

I tried to use transforms and field extraction but I couldn't make it work. I don't have ssh access so I did via the web interface

Transformation

Field extraction

What's wrong with my setup?

Thanks!

[harsmarvania57]

I have ingested below data in my lab environment with below config and it is working fine (Splunk version 7.1.2)

[A][B] blabla
[A][C] blabla

props.conf (In below config I have given DATETIME_CONFIG = CURRENT because in sample data we don't have any timestamp)

[testData]
TRANSFORMS-testProp = testData_trans
SHOULD_LINEMERGE = False
DATETIME_CONFIG = CURRENT
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

transforms.conf

[testData_trans]
DEST_KEY = MetaData:Sourcetype
REGEX = \[A\]\[B\]
FORMAT = sourcetype::testData_B

With above config [A][B] blabla line indexed with sourcetype testData_B and [A][C] blabla line indexed with sourcetype testData

[niboucher]

I tried to modify directly props.conf and transforms.conf and restart splunk but that didn't make it

transforms.conf

[testData_trans]
SOURCE_KEY = MetaData:Sourcetype
REGEX = (\[A\]\[B\].*)
FORMAT = sourcetype::testData_B

props.conf

 [sourcetype::testData]
 TRANSFORMS-testProp = testData_trans

Do you guys have any idea that could help me make this work?

[niboucher]

Thanks a lot man!
The problem was certainly caused by the lack of timestamp.
Setting it to CURRENT and using the regex you provided made it work.

Niboucher

[harsmarvania57]

Great that it worked, DATETIME_CONFIG = CURRENT assigns timestamp from splunk server itself, which means when data indexed whatever time present on splunk server that time will be assigned to events.

I have converted my comment to answers, so that you can accept & upvote it.

[harsmarvania57]

Config should be like this

props.conf

[testData]
TRANSFORMS-testProp = testData_trans

transforms.conf

[testData_trans]
DEST_KEY = MetaData:Sourcetype
REGEX = (\[A\]\[B\].*)
FORMAT = sourcetype::testData_B

[niboucher]

That didn't work either.

UF conf
inputs.conf

[monitor:///var/log/containers/*.log]
sourcetype = testData
recursive = false

Indexer conf
props.conf

 [testData]
 TRANSFORMS-testProp = testData_trans

transforms.conf

 [testData_trans]
 DEST_KEY = MetaData:Sourcetype
 REGEX = (\[A\]\[B\].*)
 FORMAT = sourcetype::testData_B

[harsmarvania57]

It should work, can you please try with REGEX = \[A\]\[B\] in transforms.conf, don't forget to restart splunk.

Additionally data which is already indexed with old sourcetype will not change only new data which will come to Indexer now onwards will have new sourcetype (Here I am guessing that UF is sending logs to Indexer directly and not via Heavy Forwarder)

[niboucher]

it didn't work

When I use regex below, it works and index everything in the dest sourcetype but adding a pattern inside the capturing group breaks it
REGEX= (.*)

[harsmarvania57]

Hi @niboucher,

Do you want to index data in different sourcetype when it comes to Indexer ? Are you running single instance, distributed environment or clustered environment ?

If you don't have ssh access to splunk server you can use REST API to do configuration on indexers but this approach is not suitable for clustered indexer, you can find Splunk SDK Python script on this answer to create props.conf configuration, similarly you can create configuration for transforms.conf (Here is another answer which will help you to do configuration which you will require to rename sourcetype based on REGEX)

[niboucher]

Hey,

Exactly!
I want data to be indexed in a different sourcetype when data arrives to the indexer.
It's a clustered environnment and I have very limited rights as a user.
REST API doesn't seem to be a suitable option.

I will deploy a new splunk instance with full admin rights to try to modify directly transforms.conf and props.conf

Thanks