Splunk Search

JSON unable to parse data into Splunk

mailtosnsolutio
Explorer

Hello Team,

Could you please help me upload this data in Splunk as I am passing into upload as JSON its unable to parse it.

2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {"pbid":313384,"pid":702,"time":1583764093000,"timestamp":"2020-03-09 14:28:13","creationTime":1583764287000,"creationTimestamp":"2020-03-09 14:31:27","name":"SaaS::Multiple SaaS Resource Deletions","components":[8352],"didRestrictions":[],"didExclusions":[],"throttle":3600,"sharedEndpoints":false,"interval":0,"sequenced":false,"active":true,"retired":false,"state":"New","commentCount":0,"triggeredComponents":[{"time":1583764092000,"timestamp":"2020-03-09 14:28:12","cbid":563836,"cid":8352,"chid":12738,"size":6,"threshold":5,"interval":1800,"logic":{"data":{"left":{"left":"A","operator":"AND","right":{"left":"C","operator":"AND","right":"D"}},"operator":"OR","right":{"left":"B","operator":"AND","right":{"left":"C","operator":"AND","right":"D"}}},"version":"v0.1"},"metric":{"mlid":306,"name":"saasrurcedeleted","label":"SaaS Resource Deleted"},"device":{"did":15727,"sid":-9,"hostname":"SaaS::Office365: cook@polar.com","firstSeen":1538579956000,"lastSeen":1583764287000,"typename":"saasprovider","typelabel":"SaaS Provider"},"triggeredFilters":[{"cfid":68892,"id":"A","filterType":"Unusual SaaS usage","comparatorType":">","arguments":{"value":40},"triggeringValue":"41"},{"cfid":68894,"id":"C","filterType":"Direction","comparatorType":"is","arguments":{"value":"in"},"triggeringValue":"in"},{"cfid":68895,"id":"D","filterType":"Message","comparatorType":"does not contain","arguments":{"value":"event=DeleteEvent"},"triggeringValue":"event=MoveToDeletedItems,user=cook@polar.com"},{"cfid":68896,"id":"d1","filterType":"Source IP","comparatorType":"display","arguments":{},"triggeringValue":"65.205.175.4"},{"cfid":68897,"id":"d2","filterType":"Message","comparatorType":"display","arguments":{},"triggeringValue":"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com"},{"cfid":68898,"id":"d3","filterType":"Unusual SaaS usage","comparatorType":"display","arguments":{},"triggeringValue":"41"},{"cfid":68899,"id":"d4","filterType":"Event details","comparatorType":"display","arguments":{},"triggeringValue":""}]}]}
Tags (4)
0 Karma
1 Solution

to4kawa
Ultra Champion
| makeresults
| eval _raw="2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {\"pbid\":313384,\"pid\":702,\"time\":1583764093000,\"timestamp\":\"2020-03-09 14:28:13\",\"creationTime\":1583764287000,\"creationTimestamp\":\"2020-03-09 14:31:27\",\"name\":\"SaaS::Multiple SaaS Resource Deletions\",\"components\":[8352],\"didRestrictions\":[],\"didExclusions\":[],\"throttle\":3600,\"sharedEndpoints\":false,\"interval\":0,\"sequenced\":false,\"active\":true,\"retired\":false,\"state\":\"New\",\"commentCount\":0,\"triggeredComponents\":[{\"time\":1583764092000,\"timestamp\":\"2020-03-09 14:28:12\",\"cbid\":563836,\"cid\":8352,\"chid\":12738,\"size\":6,\"threshold\":5,\"interval\":1800,\"logic\":{\"data\":{\"left\":{\"left\":\"A\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}},\"operator\":\"OR\",\"right\":{\"left\":\"B\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}}},\"version\":\"v0.1\"},\"metric\":{\"mlid\":306,\"name\":\"saasrurcedeleted\",\"label\":\"SaaS Resource Deleted\"},\"device\":{\"did\":15727,\"sid\":-9,\"hostname\":\"SaaS::Office365: cook@polar.com\",\"firstSeen\":1538579956000,\"lastSeen\":1583764287000,\"typename\":\"saasprovider\",\"typelabel\":\"SaaS Provider\"},\"triggeredFilters\":[{\"cfid\":68892,\"id\":\"A\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\">\",\"arguments\":{\"value\":40},\"triggeringValue\":\"41\"},{\"cfid\":68894,\"id\":\"C\",\"filterType\":\"Direction\",\"comparatorType\":\"is\",\"arguments\":{\"value\":\"in\"},\"triggeringValue\":\"in\"},{\"cfid\":68895,\"id\":\"D\",\"filterType\":\"Message\",\"comparatorType\":\"does not contain\",\"arguments\":{\"value\":\"event=DeleteEvent\"},\"triggeringValue\":\"event=MoveToDeletedItems,user=cook@polar.com\"},{\"cfid\":68896,\"id\":\"d1\",\"filterType\":\"Source IP\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"65.205.175.4\"},{\"cfid\":68897,\"id\":\"d2\",\"filterType\":\"Message\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com\"},{\"cfid\":68898,\"id\":\"d3\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"41\"},{\"cfid\":68899,\"id\":\"d4\",\"filterType\":\"Event details\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"\"}]}]}"
| rex "(?<time>\S+) (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - (?<json>{.*})" 
| rex mode=sed "s/^.*?({.*})/\1/"
| spath

This query is OK.

props.conf

TIME_FORMAT = %FT%T%:z
EXTRACT-header = ^\S+ (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - 
SEDCMD-extract_json = s/^.*?({.*})/\1/
KV_MODE = json

View solution in original post

somesoni2
Revered Legend

The data is not being parsed as JSON due to the non-json construct at the start of your event (2020-03-09T..other content... darktrace - - -.The raw data has to be pure json format in order to parsed automatically by Splunk.

0 Karma

to4kawa
Ultra Champion
| makeresults
| eval _raw="2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {\"pbid\":313384,\"pid\":702,\"time\":1583764093000,\"timestamp\":\"2020-03-09 14:28:13\",\"creationTime\":1583764287000,\"creationTimestamp\":\"2020-03-09 14:31:27\",\"name\":\"SaaS::Multiple SaaS Resource Deletions\",\"components\":[8352],\"didRestrictions\":[],\"didExclusions\":[],\"throttle\":3600,\"sharedEndpoints\":false,\"interval\":0,\"sequenced\":false,\"active\":true,\"retired\":false,\"state\":\"New\",\"commentCount\":0,\"triggeredComponents\":[{\"time\":1583764092000,\"timestamp\":\"2020-03-09 14:28:12\",\"cbid\":563836,\"cid\":8352,\"chid\":12738,\"size\":6,\"threshold\":5,\"interval\":1800,\"logic\":{\"data\":{\"left\":{\"left\":\"A\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}},\"operator\":\"OR\",\"right\":{\"left\":\"B\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}}},\"version\":\"v0.1\"},\"metric\":{\"mlid\":306,\"name\":\"saasrurcedeleted\",\"label\":\"SaaS Resource Deleted\"},\"device\":{\"did\":15727,\"sid\":-9,\"hostname\":\"SaaS::Office365: cook@polar.com\",\"firstSeen\":1538579956000,\"lastSeen\":1583764287000,\"typename\":\"saasprovider\",\"typelabel\":\"SaaS Provider\"},\"triggeredFilters\":[{\"cfid\":68892,\"id\":\"A\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\">\",\"arguments\":{\"value\":40},\"triggeringValue\":\"41\"},{\"cfid\":68894,\"id\":\"C\",\"filterType\":\"Direction\",\"comparatorType\":\"is\",\"arguments\":{\"value\":\"in\"},\"triggeringValue\":\"in\"},{\"cfid\":68895,\"id\":\"D\",\"filterType\":\"Message\",\"comparatorType\":\"does not contain\",\"arguments\":{\"value\":\"event=DeleteEvent\"},\"triggeringValue\":\"event=MoveToDeletedItems,user=cook@polar.com\"},{\"cfid\":68896,\"id\":\"d1\",\"filterType\":\"Source IP\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"65.205.175.4\"},{\"cfid\":68897,\"id\":\"d2\",\"filterType\":\"Message\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com\"},{\"cfid\":68898,\"id\":\"d3\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"41\"},{\"cfid\":68899,\"id\":\"d4\",\"filterType\":\"Event details\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"\"}]}]}"
| rex "(?<time>\S+) (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - (?<json>{.*})" 
| rex mode=sed "s/^.*?({.*})/\1/"
| spath

This query is OK.

props.conf

TIME_FORMAT = %FT%T%:z
EXTRACT-header = ^\S+ (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - 
SEDCMD-extract_json = s/^.*?({.*})/\1/
KV_MODE = json

xavierashe
Contributor

Do you have KV_mode=json set in your props.conf?

0 Karma

13tsavage
Communicator

Seeing two different _time formats here. I see 2020-03-09T 14:30:39+00:00 and then I am also seeing epoch times in "firstSeen"1538579956000, "lastSeen"1583764287000 and "time".

Is this a partial log file? Are you only trying to load this snippet of a log in to splunk or the entire log file?

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...