JSON parsing after transforms

[cornemrc]

Hello Splunk Community,

I have an issue with JSON parsing in Splunk and hope you can help me with that.

Situation:

• Logs arrive via syslog on our indexers
• Inside my app I have the following inputs.conf

[monitor:///here_is_the_correct_path]
disabled = false
host_segment = 3
index = buttercup
sourcetype = buttercup:server
host =​

• This is my props.conf

[buttercup:server]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG = 
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Logs
pulldown_type = true
TRANSFORMS-afilter = setnull, setparsing-audit, setparsing-auth
TRANSFORMS-changesourcetype = change-buttercup-server-audit, change-buttercup-server-auth

[buttercup:server:audit]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG = 
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Auditlog
pulldown_type = true
SEDCMD-strip_prefix = s/^[^{]+//g

[buttercup:server:auth]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG = 
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Authenticationlog
pulldown_type = true​

• And my transforms.conf

[change-buttercup-server-audit]
REGEX = buttercup_audit\:
FORMAT = sourcetype::buttercup:server:audit
DEST_KEY = MetaData:Sourcetype

[change-buttercup-server-auth]
REGEX = buttercup_auth\:
FORMAT = sourcetype::buttercup:server:auth
DEST_KEY = MetaData:Sourcetype

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing-audit]
REGEX = buttercup_audit\:
DEST_KEY = queue
FORMAT = indexQueue

[setparsing-auth]
REGEX = buttercup_auth\:
DEST_KEY = queue
FORMAT = indexQueue​

Description:

• After the input to index buttercup and sourcetype buttercup:server I use TRANSFORMS-afilter first, to filter everything from the syslog stream that does not include audit or auth logs. Therefore, I am using the setnull/setparsing construct in transforms.conf
• After the filtering process, data goes back into the indexQueue and I use TRANSFORMS-changesourcetype to assign the matching “buttercup:server:audit” or “buttercup:server:auth” sourcetype.
• The filtering and sourcetype assigning processes are successful, which shows me that the construct is working fine

Problem:

• The problem is that the audit log has a JSON structure which should be parsed by Splunk automatically. To achieve this, I use the SEDCMD for this sourcetype to remove the prefix in front of the JSON structure. 
• This JSON parsing is working fine, when I do a manual file input and select buttercup:server:audit directly.
• But this JSON parsing is not working for manual input when I select buttercup:server
• Therefore, it is also not working for monitor input of the syslog stream (as buttercup:server will be used first)

I think the problem has something to do with the SEDCMD and when it will be handled. Do you have any idea, how to fix that? I was thinking about doing the SEDCMD part within an additional transforms instead but I don´t know how. I have also experimented with adding INDEXED_EXTRACTIONS=JSON and KV_MODE=none (and vise versa) to the sourcetype, but no success.

[deckemha]

Hello all,

we' ve solved the problem:

Props.conf - sourcetype buttercup:server was adjusted

[buttercup:server]
TZ = UTC
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
description = buttercup Server Logs
pulldown_type = true
TRANSFORMS-afilter = setnull, setparsing-audit, setparsing-auth
TRANSFORMS-changesourcetype = change-buttercup-server-audit, change-buttercup-server-auth
TRANSFORMS-strip-front-json = strip-front-json

The last line was added and this replaced also the SEDCMD command in the buttercup:server:audit sourcetype.

transforms.conf - added the following

[strip-front-json]
REGEX = ((?<=buttercup_audit: )(.*))
FORMAT = $2
DEST_KEY = _raw

The main problem was, that after rewriting the sourcetype and filtering the events, the parsing does not have any effect anymore because it was already in the indexQueue.

SEDCMD whould have also worked in the sourcetype buttercup:server we think.

There is maybe a better/more efficient way of orderng the transforms, but it works for now.

Many Regards

Michael