Getting Data In

Help with Advanced Source Type

jaredneedell
Explorer

I'm trying to create a custom source type which is reading a TSV log file and the 3 column in the file is a JSON payload wrapped in quotes. I can't figure out how to get the source type to parse out the 3 column in a JSON format on splunk. Here's an example of a line entry below.

6680 "2020-03-06 13:50:13.254" "{"date":"3/6/2020 1:50:13 PM","received":"from FooServer (Unknown [172.20.36.5]) by smtp-dev.foo.com with ESMTP ; Fri, 6 Mar 2020 13:50:13 -0500","message-id":"id@message.com","from":"foo@thisMachine.com","recipients":"John.Smith@example.com","cc":"","subject":"Test Email"}"

Any advice would be helpful, thank you.

0 Karma
1 Solution

manjunathmeti
SplunkTrust
SplunkTrust

Add below content to transforms.conf. This will extract key value pair and index them.

transforms.conf.

[extract_json_fields]
REGEX = \"([\w-]+)\":\"([^"]*)\"
FORMAT = $1::$2

Refer extract_json_fields in props.conf under sourcetype stanza.

props.conf

[sourcetype_name]
TRANSFORMS-extract_fields = extract_json_fields

View solution in original post

woodcock
Esteemed Legend

The documentation is EXTREMELY misleading in this regard. There is no such thing as creating a sourcetype. All possible sourcetype values "already exist" in any meaningful sense of the word exist. There is NOTHING that must be done to create one before using it. Just use it and poof it now exists and works.

This is not the right way to do it, however, assuming there is no nesting and no arrays in your JSON, you can get away with doing this:

Settings -> Fields -> Field Transformations -> New Field Transformation then
Set Name to something like <YourSourcetype>_JSONpayload.
Set Type to regex-based.
Set Regular expression to (?<=[,{])"([^"]+)":"([^"]*).
Set Format to $1::$2.
Set Create multivalued fields to checked (e.g. true / yes ).
Click green Save button.

Settings -> Fields -> Field Extractions -> New Field Extraction then:
Set Name to <YourSourcetype>_JSONpayload.
Set Apply to to Sourcetype + named = <YourSourcetype>.
Set Type to Uses transform.
Set Extraction/Transform to <YourSourcetype>_JSONpayload.
Click green Save button.

to4kawa
Ultra Champion
| makeresults 
| eval _raw="6680 \"2020-03-06 13:50:13.254\" \"{\"date\":\"3/6/2020 1:50:13 PM\",\"received\":\"from FooServer (Unknown [172.20.36.5]) by smtp-dev.foo.com with ESMTP ; Fri, 6 Mar 2020 13:50:13 -0500\",\"message-id\":\" \",\"from\":\" foo@thisMachine.com\",\"recipients\":\" John.Smith@example.com\",\"cc\":\"\",\"subject\":\"Test Email\"}\"" 
| rex mode=sed "s/^.*(?={)(.*)\"/\1/g" 
| spath

this is OK.

props.conf or Create source types >> advanced

[your sourcetype]
DATETIME_CONFIG = %F %T.%3Q
SEDCMD-trim_json = s/^.*(?={)(.*)\"/\1/g
KV_MODE = json
0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Add below content to transforms.conf. This will extract key value pair and index them.

transforms.conf.

[extract_json_fields]
REGEX = \"([\w-]+)\":\"([^"]*)\"
FORMAT = $1::$2

Refer extract_json_fields in props.conf under sourcetype stanza.

props.conf

[sourcetype_name]
TRANSFORMS-extract_fields = extract_json_fields

jaredneedell
Explorer

How would I go about this if I am using the UI to create the Source Type?

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Add new setting EXTRACT-fields and put this below regex. Save and search sourcetype.

"date":"(?<date>[^\"]*)","received":"(?<received>[^\"]*)","message-id":"(?<messageid>[^\"]*)","from":"(?<from>[^\"]*)","recipients":"(?<recipients>[^\"]*)","cc":"(?<cc>[^\"]*)","subject":"(?<subject>[^\"]*)"
0 Karma
Get Updates on the Splunk Community!

Don't wait! Accept the Mission Possible: Splunk Adoption Challenge Now and Win ...

Attention everyone! We have exciting news to share! We are recruiting new members for the Mission Possible: ...

Unify Your SecOps with Splunk Mission Control

In today’s post, I'm excited to share some recent Splunk Mission Control innovations. With Splunk Mission ...

Data Preparation Made Easy: SPL2 for Edge Processor

By now, you may have heard the exciting news that Edge Processor, the easy-to-use Splunk data preparation tool ...