JSON unable to parse data into Splunk

[mailtosnsolutio]

Hello Team,

Could you please help me upload this data in Splunk as I am passing into upload as JSON its unable to parse it.

2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {"pbid":313384,"pid":702,"time":1583764093000,"timestamp":"2020-03-09 14:28:13","creationTime":1583764287000,"creationTimestamp":"2020-03-09 14:31:27","name":"SaaS::Multiple SaaS Resource Deletions","components":[8352],"didRestrictions":[],"didExclusions":[],"throttle":3600,"sharedEndpoints":false,"interval":0,"sequenced":false,"active":true,"retired":false,"state":"New","commentCount":0,"triggeredComponents":[{"time":1583764092000,"timestamp":"2020-03-09 14:28:12","cbid":563836,"cid":8352,"chid":12738,"size":6,"threshold":5,"interval":1800,"logic":{"data":{"left":{"left":"A","operator":"AND","right":{"left":"C","operator":"AND","right":"D"}},"operator":"OR","right":{"left":"B","operator":"AND","right":{"left":"C","operator":"AND","right":"D"}}},"version":"v0.1"},"metric":{"mlid":306,"name":"saasrurcedeleted","label":"SaaS Resource Deleted"},"device":{"did":15727,"sid":-9,"hostname":"SaaS::Office365: cook@polar.com","firstSeen":1538579956000,"lastSeen":1583764287000,"typename":"saasprovider","typelabel":"SaaS Provider"},"triggeredFilters":[{"cfid":68892,"id":"A","filterType":"Unusual SaaS usage","comparatorType":">","arguments":{"value":40},"triggeringValue":"41"},{"cfid":68894,"id":"C","filterType":"Direction","comparatorType":"is","arguments":{"value":"in"},"triggeringValue":"in"},{"cfid":68895,"id":"D","filterType":"Message","comparatorType":"does not contain","arguments":{"value":"event=DeleteEvent"},"triggeringValue":"event=MoveToDeletedItems,user=cook@polar.com"},{"cfid":68896,"id":"d1","filterType":"Source IP","comparatorType":"display","arguments":{},"triggeringValue":"65.205.175.4"},{"cfid":68897,"id":"d2","filterType":"Message","comparatorType":"display","arguments":{},"triggeringValue":"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com"},{"cfid":68898,"id":"d3","filterType":"Unusual SaaS usage","comparatorType":"display","arguments":{},"triggeringValue":"41"},{"cfid":68899,"id":"d4","filterType":"Event details","comparatorType":"display","arguments":{},"triggeringValue":""}]}]}

[somesoni2]

The data is not being parsed as JSON due to the non-json construct at the start of your event (2020-03-09T..other content... darktrace - - -.The raw data has to be pure json format in order to parsed automatically by Splunk.

[to4kawa · Accepted solution]

| makeresults
| eval _raw="2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {\"pbid\":313384,\"pid\":702,\"time\":1583764093000,\"timestamp\":\"2020-03-09 14:28:13\",\"creationTime\":1583764287000,\"creationTimestamp\":\"2020-03-09 14:31:27\",\"name\":\"SaaS::Multiple SaaS Resource Deletions\",\"components\":[8352],\"didRestrictions\":[],\"didExclusions\":[],\"throttle\":3600,\"sharedEndpoints\":false,\"interval\":0,\"sequenced\":false,\"active\":true,\"retired\":false,\"state\":\"New\",\"commentCount\":0,\"triggeredComponents\":[{\"time\":1583764092000,\"timestamp\":\"2020-03-09 14:28:12\",\"cbid\":563836,\"cid\":8352,\"chid\":12738,\"size\":6,\"threshold\":5,\"interval\":1800,\"logic\":{\"data\":{\"left\":{\"left\":\"A\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}},\"operator\":\"OR\",\"right\":{\"left\":\"B\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}}},\"version\":\"v0.1\"},\"metric\":{\"mlid\":306,\"name\":\"saasrurcedeleted\",\"label\":\"SaaS Resource Deleted\"},\"device\":{\"did\":15727,\"sid\":-9,\"hostname\":\"SaaS::Office365: cook@polar.com\",\"firstSeen\":1538579956000,\"lastSeen\":1583764287000,\"typename\":\"saasprovider\",\"typelabel\":\"SaaS Provider\"},\"triggeredFilters\":[{\"cfid\":68892,\"id\":\"A\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\">\",\"arguments\":{\"value\":40},\"triggeringValue\":\"41\"},{\"cfid\":68894,\"id\":\"C\",\"filterType\":\"Direction\",\"comparatorType\":\"is\",\"arguments\":{\"value\":\"in\"},\"triggeringValue\":\"in\"},{\"cfid\":68895,\"id\":\"D\",\"filterType\":\"Message\",\"comparatorType\":\"does not contain\",\"arguments\":{\"value\":\"event=DeleteEvent\"},\"triggeringValue\":\"event=MoveToDeletedItems,user=cook@polar.com\"},{\"cfid\":68896,\"id\":\"d1\",\"filterType\":\"Source IP\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"65.205.175.4\"},{\"cfid\":68897,\"id\":\"d2\",\"filterType\":\"Message\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com\"},{\"cfid\":68898,\"id\":\"d3\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"41\"},{\"cfid\":68899,\"id\":\"d4\",\"filterType\":\"Event details\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"\"}]}]}"
| rex "(?<time>\S+) (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - (?<json>{.*})" 
| rex mode=sed "s/^.*?({.*})/\1/"
| spath

This query is OK.

props.conf

TIME_FORMAT = %FT%T%:z
EXTRACT-header = ^\S+ (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - 
SEDCMD-extract_json = s/^.*?({.*})/\1/
KV_MODE = json

[xavierashe]

Do you have KV_mode=json set in your props.conf?

[13tsavage]

Seeing two different _time formats here. I see 2020-03-09T 14:30:39+00:00 and then I am also seeing epoch times in "firstSeen"1538579956000, "lastSeen"1583764287000 and "time".

Is this a partial log file? Are you only trying to load this snippet of a log in to splunk or the entire log file?