Hello Splunk Community, I have an issue with JSON parsing in Splunk and hope you can help me with that. Situation: Logs arrive via syslog on our indexers Inside my app I have the following inputs.conf [monitor:///here_is_the_correct_path]
disabled = false
host_segment = 3
index = buttercup
sourcetype = buttercup:server
host = This is my props.conf [buttercup:server]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Logs
pulldown_type = true
TRANSFORMS-afilter = setnull, setparsing-audit, setparsing-auth
TRANSFORMS-changesourcetype = change-buttercup-server-audit, change-buttercup-server-auth
[buttercup:server:audit]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Auditlog
pulldown_type = true
SEDCMD-strip_prefix = s/^[^{]+//g
[buttercup:server:auth]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Authenticationlog
pulldown_type = true And my transforms.conf [change-buttercup-server-audit]
REGEX = buttercup_audit\:
FORMAT = sourcetype::buttercup:server:audit
DEST_KEY = MetaData:Sourcetype
[change-buttercup-server-auth]
REGEX = buttercup_auth\:
FORMAT = sourcetype::buttercup:server:auth
DEST_KEY = MetaData:Sourcetype
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing-audit]
REGEX = buttercup_audit\:
DEST_KEY = queue
FORMAT = indexQueue
[setparsing-auth]
REGEX = buttercup_auth\:
DEST_KEY = queue
FORMAT = indexQueue Description: After the input to index buttercup and sourcetype buttercup:server I use TRANSFORMS-afilter first, to filter everything from the syslog stream that does not include audit or auth logs. Therefore, I am using the setnull/setparsing construct in transforms.conf After the filtering process, data goes back into the indexQueue and I use TRANSFORMS-changesourcetype to assign the matching “buttercup:server:audit” or “buttercup:server:auth” sourcetype. The filtering and sourcetype assigning processes are successful, which shows me that the construct is working fine Problem: The problem is that the audit log has a JSON structure which should be parsed by Splunk automatically. To achieve this, I use the SEDCMD for this sourcetype to remove the prefix in front of the JSON structure. This JSON parsing is working fine, when I do a manual file input and select buttercup:server:audit directly. But this JSON parsing is not working for manual input when I select buttercup:server Therefore, it is also not working for monitor input of the syslog stream (as buttercup:server will be used first) I think the problem has something to do with the SEDCMD and when it will be handled. Do you have any idea, how to fix that? I was thinking about doing the SEDCMD part within an additional transforms instead but I don´t know how. I have also experimented with adding INDEXED_EXTRACTIONS=JSON and KV_MODE=none (and vise versa) to the sourcetype, but no success.
... View more
