Getting Data In

Help with Advanced Source Type

jaredneedell
Explorer

I'm trying to create a custom source type which is reading a TSV log file and the 3 column in the file is a JSON payload wrapped in quotes. I can't figure out how to get the source type to parse out the 3 column in a JSON format on splunk. Here's an example of a line entry below.

6680 "2020-03-06 13:50:13.254" "{"date":"3/6/2020 1:50:13 PM","received":"from FooServer (Unknown [172.20.36.5]) by smtp-dev.foo.com with ESMTP ; Fri, 6 Mar 2020 13:50:13 -0500","message-id":"id@message.com","from":"foo@thisMachine.com","recipients":"John.Smith@example.com","cc":"","subject":"Test Email"}"

Any advice would be helpful, thank you.

0 Karma
1 Solution

manjunathmeti
SplunkTrust
SplunkTrust

Add below content to transforms.conf. This will extract key value pair and index them.

transforms.conf.

[extract_json_fields]
REGEX = \"([\w-]+)\":\"([^"]*)\"
FORMAT = $1::$2

Refer extract_json_fields in props.conf under sourcetype stanza.

props.conf

[sourcetype_name]
TRANSFORMS-extract_fields = extract_json_fields

View solution in original post

woodcock
Esteemed Legend

The documentation is EXTREMELY misleading in this regard. There is no such thing as creating a sourcetype. All possible sourcetype values "already exist" in any meaningful sense of the word exist. There is NOTHING that must be done to create one before using it. Just use it and poof it now exists and works.

This is not the right way to do it, however, assuming there is no nesting and no arrays in your JSON, you can get away with doing this:

Settings -> Fields -> Field Transformations -> New Field Transformation then
Set Name to something like <YourSourcetype>_JSONpayload.
Set Type to regex-based.
Set Regular expression to (?<=[,{])"([^"]+)":"([^"]*).
Set Format to $1::$2.
Set Create multivalued fields to checked (e.g. true / yes ).
Click green Save button.

Settings -> Fields -> Field Extractions -> New Field Extraction then:
Set Name to <YourSourcetype>_JSONpayload.
Set Apply to to Sourcetype + named = <YourSourcetype>.
Set Type to Uses transform.
Set Extraction/Transform to <YourSourcetype>_JSONpayload.
Click green Save button.

to4kawa
Ultra Champion
| makeresults 
| eval _raw="6680 \"2020-03-06 13:50:13.254\" \"{\"date\":\"3/6/2020 1:50:13 PM\",\"received\":\"from FooServer (Unknown [172.20.36.5]) by smtp-dev.foo.com with ESMTP ; Fri, 6 Mar 2020 13:50:13 -0500\",\"message-id\":\" \",\"from\":\" foo@thisMachine.com\",\"recipients\":\" John.Smith@example.com\",\"cc\":\"\",\"subject\":\"Test Email\"}\"" 
| rex mode=sed "s/^.*(?={)(.*)\"/\1/g" 
| spath

this is OK.

props.conf or Create source types >> advanced

[your sourcetype]
DATETIME_CONFIG = %F %T.%3Q
SEDCMD-trim_json = s/^.*(?={)(.*)\"/\1/g
KV_MODE = json
0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Add below content to transforms.conf. This will extract key value pair and index them.

transforms.conf.

[extract_json_fields]
REGEX = \"([\w-]+)\":\"([^"]*)\"
FORMAT = $1::$2

Refer extract_json_fields in props.conf under sourcetype stanza.

props.conf

[sourcetype_name]
TRANSFORMS-extract_fields = extract_json_fields

jaredneedell
Explorer

How would I go about this if I am using the UI to create the Source Type?

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Add new setting EXTRACT-fields and put this below regex. Save and search sourcetype.

"date":"(?<date>[^\"]*)","received":"(?<received>[^\"]*)","message-id":"(?<messageid>[^\"]*)","from":"(?<from>[^\"]*)","recipients":"(?<recipients>[^\"]*)","cc":"(?<cc>[^\"]*)","subject":"(?<subject>[^\"]*)"
0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Spotlight on Security: Adoption Motions, War Stories, and More

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...