Using setnull and setparsing for two different sourcetypes

inderjot_rasila — Wed, 30 Sep 2020 00:04:32 GMT

Hello Everyone,

We have following props.conf

[sourcetypeA]
KV_MODE = json
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q%Z
TRUNCATE = 0
LINE_BREAKER = ([\n\r]+){
TIME_PREFIX = (\"timestamp\"[^\"]+\")
TRANSFORMS-set = setnull,setparsing

and transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = Regex1
DEST_KEY = queue
FORMAT = indexQueue

Using this configuration we are getting filtered data in splunk and it is working as expected.

No we have a requirement where we want to apply similar settings to another sourcetype say sourcetypeB with having different regex for [setparsing].

I have updated the props.conf as

[sourcetypeB]
KV_MODE = json
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3Q%Z
TRUNCATE = 0
LINE_BREAKER = ([\n\r]+){
TIME_PREFIX = (\"timestamp\"[^\"]+\")
TRANSFORMS-set = setnull,setparsing1

Transforms.conf has been modified as:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = Regex1
DEST_KEY = queue
FORMAT = indexQueue

[setparsing1]
REGEX = Regex2
DEST_KEY = queue
FORMAT = indexQueue

After applying these setting I see data only getting indexed for SourcetypeB and no data for SourcetypeA.

Could anyone please help what do i need to change to get data for both sourcetypes A and B. I have tried multiple combinations but only getting data for one sourcetype at one time

Regards,
Inderjot

Re: Using setnull and setparsing for two different sourcetypes

woodcock — Thu, 11 Apr 2019 00:09:53 GMT

The only thing that makes sense is that your Rregex1 is incorrect and never matches.

Re: Using setnull and setparsing for two different sourcetypes

inderjot_rasila — Thu, 11 Apr 2019 17:21:42 GMT

Regex1 is just a sample here we have actual application name which matches but only data from one sourcetype gets ingested at once

Re: Using setnull and setparsing for two different sourcetypes

gurlest — Thu, 30 May 2019 19:40:44 GMT

The only thing I can think to try would be to have two totally different, yet exactly the same 'setnull' stanzas in transforms.conf > setnullA for sourcetypeA and setnullB for sourcetypeB each with their own 'setparsing' as you have done already.

topic Using setnull and setparsing for two different sourcetypes in Getting Data In

Using setnull and setparsing for two different sourcetypes

Re: Using setnull and setparsing for two different sourcetypes

Re: Using setnull and setparsing for two different sourcetypes

Re: Using setnull and setparsing for two different sourcetypes