topic Re: Change sourcetype via field extraction and transforms in Getting Data In

Change sourcetype via field extraction and transforms

niboucher — Tue, 02 Oct 2018 09:56:13 GMT

Hi there,

One of UF is configured to send logs to sourcetype testData.
I'd like to push some of those logs matching a certain pattern (all logs matching the "[A][B]" pattern) to sourcetype testData_B.

Sample of log

[A][B] blabla
[A][C] blabla

I tried to use transforms and field extraction but I couldn't make it work. I don't have ssh access so I did via the web interface

Transformation

Field extraction

What's wrong with my setup?

Thanks!

Re: Change sourcetype via field extraction and transforms

harsmarvania57 — Tue, 02 Oct 2018 12:59:49 GMT

Hi @niboucher,

Do you want to index data in different sourcetype when it comes to Indexer ? Are you running single instance, distributed environment or clustered environment ?

If you don't have ssh access to splunk server you can use REST API to do configuration on indexers but this approach is not suitable for clustered indexer, you can find Splunk SDK Python script on this answer to create props.conf configuration, similarly you can create configuration for transforms.conf (Here is another answer which will help you to do configuration which you will require to rename sourcetype based on REGEX)

Re: Change sourcetype via field extraction and transforms

niboucher — Tue, 02 Oct 2018 13:44:56 GMT

Hey,

Exactly!
I want data to be indexed in a different sourcetype when data arrives to the indexer.
It's a clustered environnment and I have very limited rights as a user.
REST API doesn't seem to be a suitable option.

I will deploy a new splunk instance with full admin rights to try to modify directly transforms.conf and props.conf

Thanks

Re: Change sourcetype via field extraction and transforms

niboucher — Tue, 02 Oct 2018 16:14:04 GMT

I tried to modify directly props.conf and transforms.conf and restart splunk but that didn't make it

transforms.conf

[testData_trans]
SOURCE_KEY = MetaData:Sourcetype
REGEX = (\[A\]\[B\].*)
FORMAT = sourcetype::testData_B

props.conf

 [sourcetype::testData]
 TRANSFORMS-testProp = testData_trans

Do you guys have any idea that could help me make this work?

Re: Change sourcetype via field extraction and transforms

harsmarvania57 — Tue, 02 Oct 2018 16:36:08 GMT

Config should be like this

props.conf

[testData]
TRANSFORMS-testProp = testData_trans

transforms.conf

[testData_trans]
DEST_KEY = MetaData:Sourcetype
REGEX = (\[A\]\[B\].*)
FORMAT = sourcetype::testData_B

Re: Change sourcetype via field extraction and transforms

niboucher — Wed, 03 Oct 2018 07:56:39 GMT

That didn't work either.

UF conf
inputs.conf

[monitor:///var/log/containers/*.log]
sourcetype = testData
recursive = false

Indexer conf
props.conf

 [testData]
 TRANSFORMS-testProp = testData_trans

transforms.conf

 [testData_trans]
 DEST_KEY = MetaData:Sourcetype
 REGEX = (\[A\]\[B\].*)
 FORMAT = sourcetype::testData_B

Re: Change sourcetype via field extraction and transforms

harsmarvania57 — Wed, 03 Oct 2018 08:15:10 GMT

It should work, can you please try with REGEX = \[A\]\[B\] in transforms.conf, don't forget to restart splunk.

Additionally data which is already indexed with old sourcetype will not change only new data which will come to Indexer now onwards will have new sourcetype (Here I am guessing that UF is sending logs to Indexer directly and not via Heavy Forwarder)

Re: Change sourcetype via field extraction and transforms

niboucher — Thu, 04 Oct 2018 07:45:23 GMT

it didn't work

When I use regex below, it works and index everything in the dest sourcetype but adding a pattern inside the capturing group breaks it
REGEX= (.*)

Re: Change sourcetype via field extraction and transforms

harsmarvania57 — Thu, 04 Oct 2018 08:48:26 GMT

I have ingested below data in my lab environment with below config and it is working fine (Splunk version 7.1.2)

[A][B] blabla
[A][C] blabla

props.conf (In below config I have given DATETIME_CONFIG = CURRENT because in sample data we don't have any timestamp)

[testData]
TRANSFORMS-testProp = testData_trans
SHOULD_LINEMERGE = False
DATETIME_CONFIG = CURRENT
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

transforms.conf

[testData_trans]
DEST_KEY = MetaData:Sourcetype
REGEX = \[A\]\[B\]
FORMAT = sourcetype::testData_B

With above config [A][B] blabla line indexed with sourcetype testData_B and [A][C] blabla line indexed with sourcetype testData

Re: Change sourcetype via field extraction and transforms

niboucher — Thu, 04 Oct 2018 16:59:08 GMT

Thanks a lot man!
The problem was certainly caused by the lack of timestamp.
Setting it to CURRENT and using the regex you provided made it work.

Niboucher

Re: Change sourcetype via field extraction and transforms

harsmarvania57 — Thu, 04 Oct 2018 18:24:02 GMT

Great that it worked, DATETIME_CONFIG = CURRENT assigns timestamp from splunk server itself, which means when data indexed whatever time present on splunk server that time will be assigned to events.

I have converted my comment to answers, so that you can accept & upvote it.