Help with passing variable/sourcetype from props.conf to transforms.conf

damucka — Tue, 29 Sep 2020 23:05:00 GMT

Hello,

I am working to make my props.conf and transforms.conf more general / working for the new sourcetypes without the necessity to change it.
On the example below, for the [(?::){0}_hanatraces] I would like to pass the sourcetype name from the props.conf to transforms.conf parameterChangelog_clone and there I would like the new sourcetype with the prefix (?::){0} to be created / cloned from the original.
How would I do this?

At the moment both config files look as follows.

props.conf:

[default]
TRUNCATE = 0

[(?::){0}*_hanatraces]
TRANSFORMS-parameterChangelog_clone
TRANSFORMS-eliminatedebug = setnull

[(?::){0}*_*statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

[(?::){0}*_transports]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
MAX_TIMESTAMP_LOOKAHEAD=36
LINE_BREAKER= (1\sETP199\s#*)
TIME_PREFIX = 1\sETP110\send\sdate\sand\stime\s*:\s*\"(\d+)
TIME_FORMAT = %Y%m%d%H%M%S

[(?::){0}*_crashdumps]
BREAK_ONLY_BEFORE_DATE=false
CHARSET=UTF-8
LINE_BREAKER=([\r\n+])\[\w+\]\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S %3N
TIME_PREFIX=(?:\[\w+\]\s+\w+(?:\s+\w+)*?\:\s\(|Exception\stime\:\s)
disabled=false 

[(?::){0}*_rtedumps]
BREAK_ONLY_BEFORE_DATE=false
CHARSET=UTF-8
LINE_BREAKER=([\r\n+])\[\w+\]\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S %3N
TIME_PREFIX=(?:\[\w+\]\s+\w+(?:\s+\w+)*?\:\s\(|Runtime\sdump\stime\:\s)
disabled=false

transforms.conf:

################# Clone the parameter changes entries to the mlbso_changelog index, new sourcetype SID_parameterChangelog
[parameterChangelog_clone]
 CLONE_SOURCETYPE = (?::){0}*_parameterChangelog
 REGEX            = .*(?i)(csns\.admin\.AdminConnection|csns\.admin\.commands|alter system alter configuration)(?-i).*
 DEST_KEY         = _MetaData:Index
 FORMAT           = mlbso_changelog 
#################

################# Get rid of the "d" - debug loglevel entries from the logs 
[setnull]
 REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\s[dD]
 DEST_KEY=queue
 FORMAT=nullQueue
#################

################# Mask the D/C/I-user names and the IP-Addresses
[anonymize-user]
REGEX = ([=,>'\\":;|\s])([ICDicd]\d{3,})([,<:;|'&\\"\s])
FORMAT = $1(D\/C\/I)###########$3
DEST_KEY = _raw
REPEAT_MATCH = true

[anonymize-ip]
REGEX = ([=,\s])(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})([,:\s])
FORMAT = $1##.##.##.##$3
DEST_KEY = _raw
REPEAT_MATCH = true
#################

Re: Help with passing variable/sourcetype from props.conf to transforms.conf

maciep — Thu, 07 Feb 2019 02:58:51 GMT

i don't think that's a supported use of the sourcetype stanza, so i doubt you can use something similar in transforms. I wonder if you could maybe create a new field set to the sourcetype, then use that later to set it?

Not even remotely tested...just an idea that probably won't work 🙂

transforms

[save_sourcetype]
REGEX = ^(.*)$
FORMAT = new_st::$1_parameterChangelog
SOURCE_KEY = MetaData:Sourcetype
WRITE_META = true

[parameterChangelog_clone]
CLONE_SOURCETYPE = temp:clone
REGEX = .*(?i)(csns\.admin\.AdminConnection|csns\.admin\.commands|alter system alter configuration)(?-i).*
DEST_KEY = _MetaData:Index
FORMAT = mlbso_changelog 

[set_sourcetype]
REGEX = ^(.*)$
SOURCE_KEY = field:new_st
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

props

[(?::){0}*_hanatraces]
 TRANSFORMS-clone = save_sourcetype, parameterChangelog_clone
 TRANSFORMS-eliminatedebug = setnull

[temp:clone]
TRANSFORMS-set_sourcetype = set_sourcetype

topic Re: Help with passing variable/sourcetype from props.conf to transforms.conf in Getting Data In

Help with passing variable/sourcetype from props.conf to transforms.conf

Re: Help with passing variable/sourcetype from props.conf to transforms.conf