JSON parsing after transforms

cornemrc — Mon, 10 May 2021 06:57:51 GMT

Hello Splunk Community,

I have an issue with JSON parsing in Splunk and hope you can help me with that.

Situation:

Logs arrive via syslog on our indexers
Inside my app I have the following inputs.conf

[monitor:///here_is_the_correct_path] disabled = false host_segment = 3 index = buttercup sourcetype = buttercup:server host =

This is my props.conf

[buttercup:server] TZ = UTC TIME_FORMAT = %Y-%m-%dT%H:%M:%S DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom description = buttercup Server Logs pulldown_type = true TRANSFORMS-afilter = setnull, setparsing-audit, setparsing-auth TRANSFORMS-changesourcetype = change-buttercup-server-audit, change-buttercup-server-auth [buttercup:server:audit] TZ = UTC TIME_FORMAT = %Y-%m-%dT%H:%M:%S DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom description = buttercup Server Auditlog pulldown_type = true SEDCMD-strip_prefix = s/^[^{]+//g [buttercup:server:auth] TZ = UTC TIME_FORMAT = %Y-%m-%dT%H:%M:%S DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom description = buttercup Server Authenticationlog pulldown_type = true

And my transforms.conf

[change-buttercup-server-audit] REGEX = buttercup_audit\: FORMAT = sourcetype::buttercup:server:audit DEST_KEY = MetaData:Sourcetype [change-buttercup-server-auth] REGEX = buttercup_auth\: FORMAT = sourcetype::buttercup:server:auth DEST_KEY = MetaData:Sourcetype [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing-audit] REGEX = buttercup_audit\: DEST_KEY = queue FORMAT = indexQueue [setparsing-auth] REGEX = buttercup_auth\: DEST_KEY = queue FORMAT = indexQueue

Description:

After the input to index buttercup and sourcetype buttercup:server I use TRANSFORMS-afilter first, to filter everything from the syslog stream that does not include audit or auth logs. Therefore, I am using the setnull/setparsing construct in transforms.conf
After the filtering process, data goes back into the indexQueue and I use TRANSFORMS-changesourcetype to assign the matching “buttercup:server:audit” or “buttercup:server:auth” sourcetype.
The filtering and sourcetype assigning processes are successful, which shows me that the construct is working fine

Problem:

The problem is that the audit log has a JSON structure which should be parsed by Splunk automatically. To achieve this, I use the SEDCMD for this sourcetype to remove the prefix in front of the JSON structure.
This JSON parsing is working fine, when I do a manual file input and select buttercup:server:audit directly.
But this JSON parsing is not working for manual input when I select buttercup:server
Therefore, it is also not working for monitor input of the syslog stream (as buttercup:server will be used first)

I think the problem has something to do with the SEDCMD and when it will be handled. Do you have any idea, how to fix that? I was thinking about doing the SEDCMD part within an additional transforms instead but I don´t know how. I have also experimented with adding INDEXED_EXTRACTIONS=JSON and KV_MODE=none (and vise versa) to the sourcetype, but no success.

Re: JSON parsing after transforms

deckemha — Mon, 17 May 2021 13:11:21 GMT

Hello all,

we' ve solved the problem:

Props.conf - sourcetype buttercup:server was adjusted

The last line was added and this replaced also the SEDCMD command in the buttercup:server:audit sourcetype.

transforms.conf - added the following

[strip-front-json] REGEX = ((?<=buttercup_audit: )(.*)) FORMAT = $2 DEST_KEY = _raw

The main problem was, that after rewriting the sourcetype and filtering the events, the parsing does not have any effect anymore because it was already in the indexQueue.

SEDCMD whould have also worked in the sourcetype buttercup:server we think.

There is maybe a better/more efficient way of orderng the transforms, but it works for now.

Many Regards

Michael

topic JSON parsing after transforms in Getting Data In

JSON parsing after transforms

Re: JSON parsing after transforms