Splunk Search

JSON unable to parse data into Splunk

mailtosnsolutio
Explorer

Hello Team,

Could you please help me upload this data in Splunk as I am passing into upload as JSON its unable to parse it.

2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {"pbid":313384,"pid":702,"time":1583764093000,"timestamp":"2020-03-09 14:28:13","creationTime":1583764287000,"creationTimestamp":"2020-03-09 14:31:27","name":"SaaS::Multiple SaaS Resource Deletions","components":[8352],"didRestrictions":[],"didExclusions":[],"throttle":3600,"sharedEndpoints":false,"interval":0,"sequenced":false,"active":true,"retired":false,"state":"New","commentCount":0,"triggeredComponents":[{"time":1583764092000,"timestamp":"2020-03-09 14:28:12","cbid":563836,"cid":8352,"chid":12738,"size":6,"threshold":5,"interval":1800,"logic":{"data":{"left":{"left":"A","operator":"AND","right":{"left":"C","operator":"AND","right":"D"}},"operator":"OR","right":{"left":"B","operator":"AND","right":{"left":"C","operator":"AND","right":"D"}}},"version":"v0.1"},"metric":{"mlid":306,"name":"saasrurcedeleted","label":"SaaS Resource Deleted"},"device":{"did":15727,"sid":-9,"hostname":"SaaS::Office365: cook@polar.com","firstSeen":1538579956000,"lastSeen":1583764287000,"typename":"saasprovider","typelabel":"SaaS Provider"},"triggeredFilters":[{"cfid":68892,"id":"A","filterType":"Unusual SaaS usage","comparatorType":">","arguments":{"value":40},"triggeringValue":"41"},{"cfid":68894,"id":"C","filterType":"Direction","comparatorType":"is","arguments":{"value":"in"},"triggeringValue":"in"},{"cfid":68895,"id":"D","filterType":"Message","comparatorType":"does not contain","arguments":{"value":"event=DeleteEvent"},"triggeringValue":"event=MoveToDeletedItems,user=cook@polar.com"},{"cfid":68896,"id":"d1","filterType":"Source IP","comparatorType":"display","arguments":{},"triggeringValue":"65.205.175.4"},{"cfid":68897,"id":"d2","filterType":"Message","comparatorType":"display","arguments":{},"triggeringValue":"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com"},{"cfid":68898,"id":"d3","filterType":"Unusual SaaS usage","comparatorType":"display","arguments":{},"triggeringValue":"41"},{"cfid":68899,"id":"d4","filterType":"Event details","comparatorType":"display","arguments":{},"triggeringValue":""}]}]}
Tags (4)
0 Karma
1 Solution

to4kawa
Ultra Champion
| makeresults
| eval _raw="2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {\"pbid\":313384,\"pid\":702,\"time\":1583764093000,\"timestamp\":\"2020-03-09 14:28:13\",\"creationTime\":1583764287000,\"creationTimestamp\":\"2020-03-09 14:31:27\",\"name\":\"SaaS::Multiple SaaS Resource Deletions\",\"components\":[8352],\"didRestrictions\":[],\"didExclusions\":[],\"throttle\":3600,\"sharedEndpoints\":false,\"interval\":0,\"sequenced\":false,\"active\":true,\"retired\":false,\"state\":\"New\",\"commentCount\":0,\"triggeredComponents\":[{\"time\":1583764092000,\"timestamp\":\"2020-03-09 14:28:12\",\"cbid\":563836,\"cid\":8352,\"chid\":12738,\"size\":6,\"threshold\":5,\"interval\":1800,\"logic\":{\"data\":{\"left\":{\"left\":\"A\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}},\"operator\":\"OR\",\"right\":{\"left\":\"B\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}}},\"version\":\"v0.1\"},\"metric\":{\"mlid\":306,\"name\":\"saasrurcedeleted\",\"label\":\"SaaS Resource Deleted\"},\"device\":{\"did\":15727,\"sid\":-9,\"hostname\":\"SaaS::Office365: cook@polar.com\",\"firstSeen\":1538579956000,\"lastSeen\":1583764287000,\"typename\":\"saasprovider\",\"typelabel\":\"SaaS Provider\"},\"triggeredFilters\":[{\"cfid\":68892,\"id\":\"A\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\">\",\"arguments\":{\"value\":40},\"triggeringValue\":\"41\"},{\"cfid\":68894,\"id\":\"C\",\"filterType\":\"Direction\",\"comparatorType\":\"is\",\"arguments\":{\"value\":\"in\"},\"triggeringValue\":\"in\"},{\"cfid\":68895,\"id\":\"D\",\"filterType\":\"Message\",\"comparatorType\":\"does not contain\",\"arguments\":{\"value\":\"event=DeleteEvent\"},\"triggeringValue\":\"event=MoveToDeletedItems,user=cook@polar.com\"},{\"cfid\":68896,\"id\":\"d1\",\"filterType\":\"Source IP\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"65.205.175.4\"},{\"cfid\":68897,\"id\":\"d2\",\"filterType\":\"Message\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com\"},{\"cfid\":68898,\"id\":\"d3\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"41\"},{\"cfid\":68899,\"id\":\"d4\",\"filterType\":\"Event details\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"\"}]}]}"
| rex "(?<time>\S+) (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - (?<json>{.*})" 
| rex mode=sed "s/^.*?({.*})/\1/"
| spath

This query is OK.

props.conf

TIME_FORMAT = %FT%T%:z
EXTRACT-header = ^\S+ (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - 
SEDCMD-extract_json = s/^.*?({.*})/\1/
KV_MODE = json

View solution in original post

somesoni2
Revered Legend

The data is not being parsed as JSON due to the non-json construct at the start of your event (2020-03-09T..other content... darktrace - - -.The raw data has to be pure json format in order to parsed automatically by Splunk.

0 Karma

to4kawa
Ultra Champion
| makeresults
| eval _raw="2020-03-09T14:30:39+00:00 172.20.0.20 2020-03-09T14:28:13Z cos-darktrace.polarisalpha.com darktrace - - - {\"pbid\":313384,\"pid\":702,\"time\":1583764093000,\"timestamp\":\"2020-03-09 14:28:13\",\"creationTime\":1583764287000,\"creationTimestamp\":\"2020-03-09 14:31:27\",\"name\":\"SaaS::Multiple SaaS Resource Deletions\",\"components\":[8352],\"didRestrictions\":[],\"didExclusions\":[],\"throttle\":3600,\"sharedEndpoints\":false,\"interval\":0,\"sequenced\":false,\"active\":true,\"retired\":false,\"state\":\"New\",\"commentCount\":0,\"triggeredComponents\":[{\"time\":1583764092000,\"timestamp\":\"2020-03-09 14:28:12\",\"cbid\":563836,\"cid\":8352,\"chid\":12738,\"size\":6,\"threshold\":5,\"interval\":1800,\"logic\":{\"data\":{\"left\":{\"left\":\"A\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}},\"operator\":\"OR\",\"right\":{\"left\":\"B\",\"operator\":\"AND\",\"right\":{\"left\":\"C\",\"operator\":\"AND\",\"right\":\"D\"}}},\"version\":\"v0.1\"},\"metric\":{\"mlid\":306,\"name\":\"saasrurcedeleted\",\"label\":\"SaaS Resource Deleted\"},\"device\":{\"did\":15727,\"sid\":-9,\"hostname\":\"SaaS::Office365: cook@polar.com\",\"firstSeen\":1538579956000,\"lastSeen\":1583764287000,\"typename\":\"saasprovider\",\"typelabel\":\"SaaS Provider\"},\"triggeredFilters\":[{\"cfid\":68892,\"id\":\"A\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\">\",\"arguments\":{\"value\":40},\"triggeringValue\":\"41\"},{\"cfid\":68894,\"id\":\"C\",\"filterType\":\"Direction\",\"comparatorType\":\"is\",\"arguments\":{\"value\":\"in\"},\"triggeringValue\":\"in\"},{\"cfid\":68895,\"id\":\"D\",\"filterType\":\"Message\",\"comparatorType\":\"does not contain\",\"arguments\":{\"value\":\"event=DeleteEvent\"},\"triggeringValue\":\"event=MoveToDeletedItems,user=cook@polar.com\"},{\"cfid\":68896,\"id\":\"d1\",\"filterType\":\"Source IP\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"65.205.175.4\"},{\"cfid\":68897,\"id\":\"d2\",\"filterType\":\"Message\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"event=MoveToDeletedItems,user=michael.cook@polarisalpha.com\"},{\"cfid\":68898,\"id\":\"d3\",\"filterType\":\"Unusual SaaS usage\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"41\"},{\"cfid\":68899,\"id\":\"d4\",\"filterType\":\"Event details\",\"comparatorType\":\"display\",\"arguments\":{},\"triggeringValue\":\"\"}]}]}"
| rex "(?<time>\S+) (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - (?<json>{.*})" 
| rex mode=sed "s/^.*?({.*})/\1/"
| spath

This query is OK.

props.conf

TIME_FORMAT = %FT%T%:z
EXTRACT-header = ^\S+ (?<ip>\S+) (?<stime>\S+) (?<site>\S+) (?<software>\S+) - - - 
SEDCMD-extract_json = s/^.*?({.*})/\1/
KV_MODE = json

xavierashe
Contributor

Do you have KV_mode=json set in your props.conf?

0 Karma

13tsavage
Communicator

Seeing two different _time formats here. I see 2020-03-09T 14:30:39+00:00 and then I am also seeing epoch times in "firstSeen"1538579956000, "lastSeen"1583764287000 and "time".

Is this a partial log file? Are you only trying to load this snippet of a log in to splunk or the entire log file?

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...