Getting Data In

Help with passing variable/sourcetype from props.conf to transforms.conf

damucka
Builder

Hello,

I am working to make my props.conf and transforms.conf more general / working for the new sourcetypes without the necessity to change it.
On the example below, for the [(?::){0}_hanatraces] I would like to pass the sourcetype name from the props.conf to transforms.conf parameterChangelog_clone and there I would like the new sourcetype with the prefix (?::){0} to be created / cloned from the original.
How would I do this?

At the moment both config files look as follows.

props.conf:

[default]
TRUNCATE = 0

[(?::){0}*_hanatraces]
TRANSFORMS-parameterChangelog_clone
TRANSFORMS-eliminatedebug = setnull

[(?::){0}*_*statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

[(?::){0}*_transports]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
MAX_TIMESTAMP_LOOKAHEAD=36
LINE_BREAKER= (1\sETP199\s#*)
TIME_PREFIX = 1\sETP110\send\sdate\sand\stime\s*:\s*\"(\d+)
TIME_FORMAT = %Y%m%d%H%M%S

[(?::){0}*_crashdumps]
BREAK_ONLY_BEFORE_DATE=false
CHARSET=UTF-8
LINE_BREAKER=([\r\n+])\[\w+\]\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S %3N
TIME_PREFIX=(?:\[\w+\]\s+\w+(?:\s+\w+)*?\:\s\(|Exception\stime\:\s)
disabled=false 

[(?::){0}*_rtedumps]
BREAK_ONLY_BEFORE_DATE=false
CHARSET=UTF-8
LINE_BREAKER=([\r\n+])\[\w+\]\s+\w+
MAX_TIMESTAMP_LOOKAHEAD=23
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S %3N
TIME_PREFIX=(?:\[\w+\]\s+\w+(?:\s+\w+)*?\:\s\(|Runtime\sdump\stime\:\s)
disabled=false

transforms.conf:

################# Clone the parameter changes entries to the mlbso_changelog index, new sourcetype SID_parameterChangelog
[parameterChangelog_clone]
 CLONE_SOURCETYPE = (?::){0}*_parameterChangelog
 REGEX            = .*(?i)(csns\.admin\.AdminConnection|csns\.admin\.commands|alter system alter configuration)(?-i).*
 DEST_KEY         = _MetaData:Index
 FORMAT           = mlbso_changelog 
#################

################# Get rid of the "d" - debug loglevel entries from the logs 
[setnull]
 REGEX=(?m)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}\s[dD]
 DEST_KEY=queue
 FORMAT=nullQueue
#################

################# Mask the D/C/I-user names and the IP-Addresses
[anonymize-user]
REGEX = ([=,>'\\":;|\s])([ICDicd]\d{3,})([,<:;|'&\\"\s])
FORMAT = $1(D\/C\/I)###########$3
DEST_KEY = _raw
REPEAT_MATCH = true

[anonymize-ip]
REGEX = ([=,\s])(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})([,:\s])
FORMAT = $1##.##.##.##$3
DEST_KEY = _raw
REPEAT_MATCH = true
#################
Tags (1)
0 Karma

maciep
Champion

i don't think that's a supported use of the sourcetype stanza, so i doubt you can use something similar in transforms. I wonder if you could maybe create a new field set to the sourcetype, then use that later to set it?

Not even remotely tested...just an idea that probably won't work :slightly_smiling_face:

transforms

[save_sourcetype]
REGEX = ^(.*)$
FORMAT = new_st::$1_parameterChangelog
SOURCE_KEY = MetaData:Sourcetype
WRITE_META = true

[parameterChangelog_clone]
CLONE_SOURCETYPE = temp:clone
REGEX = .*(?i)(csns\.admin\.AdminConnection|csns\.admin\.commands|alter system alter configuration)(?-i).*
DEST_KEY = _MetaData:Index
FORMAT = mlbso_changelog 

[set_sourcetype]
REGEX = ^(.*)$
SOURCE_KEY = field:new_st
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

props

[(?::){0}*_hanatraces]
 TRANSFORMS-clone = save_sourcetype, parameterChangelog_clone
 TRANSFORMS-eliminatedebug = setnull

[temp:clone]
TRANSFORMS-set_sourcetype = set_sourcetype
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...