topic Re: About Parsing JSON Log in splunk in Splunk Search

About Parsing JSON Log in splunk

huylbq — Fri, 18 Aug 2023 07:08:46 GMT

<6>2023-08-17T04:51:52Z 49786672a6c4 PICUS[1]: {"common":{"unique_id":"6963f063-a68d-482c-a22a-9e96ada33126","time":"2023-08-17T04:51:51.668553048Z","type":"","action":"","user_id":0,"user_email":"","user_first_name":"","user_last_name":"","account_id":7161,"ip":"","done_with_api":false,"platform_licences":null},"data":{"ActionID":26412,"ActionName":"Zebrocy Malware Downloader used by APT28 Threat Group .EXE File Download Variant-3","AgentName":"VICTIM-99","AssessmentName":"LAB02","CVE":"_","DestinationPort":"443","File":"682822.exe","Hash":"eb81c1be62f23ac7700c70d866e84f5bc354f88e6f7d84fd65374f84e252e76b","Result":{"alert_result":"","has_detection_result":false,"logging_result":"","prevention_result":"blocked"},"RunID":109802,"SimulationID":36236,"SourcePort":"51967","Time":5}}

I have a raw log like that, can you help me to parsing it into seperated lines ?

Re: About Parsing JSON Log in splunk

gcusello — Fri, 18 Aug 2023 07:15:10 GMT

Hi @huylbq,

usually json files aren't divided in separated lines because there's an header commo to more definitions.

You can extract all the fields using the "INDEXED_EXTRACTIONS = json" option in the sourcetype or the "spath" command (https://community.splunk.com/t5/Splunk-Enterprise/spath-command/m-p/518343).

Ciao.

Giuseppe

Re: About Parsing JSON Log in splunk

yuanliu — Fri, 18 Aug 2023 07:45:29 GMT

Like @gcusello said, you don't need to parse raw logs into separate lines. You just need to extract the part that is compliant JSON, then use spath to extract JSON nodes into Splunk fields.

| eval json = replace(_raw, "^[^\{]+", "") | spath input=json

Your sample event gives

common.account_id	common.action	common.done_with_api	...	data.Time	json
7161		false		5	{"common":{"unique_id":"6963f063-a68d-482c-a22a-9e96ada33126","time":"2023-08-17T04:51:51.668553048Z","type":"","action":"","user_id":0,"user_email":"","user_first_name":"","user_last_name":"","account_id":7161,"ip":"","done_with_api":false,"platform_licences":null},"data":{"ActionID":26412,"ActionName":"Zebrocy Malware Downloader used by APT28 Threat Group .EXE File Download Variant-3","AgentName":"VICTIM-99","AssessmentName":"LAB02","CVE":"_","DestinationPort":"443","File":"682822.exe","Hash":"eb81c1be62f23ac7700c70d866e84f5bc354f88e6f7d84fd65374f84e252e76b","Result":{"alert_result":"","has_detection_result":false,"logging_result":"","prevention_result":"blocked"},"RunID":109802,"SimulationID":36236,"SourcePort":"51967","Time":5}}

Here is an emulation you can play with and compare with real data

| makeresults | eval _raw = "<6>2023-08-17T04:51:52Z 49786672a6c4 PICUS[1]: {\"common\":{\"unique_id\":\"6963f063-a68d-482c-a22a-9e96ada33126\",\"time\":\"2023-08-17T04:51:51.668553048Z\",\"type\":\"\",\"action\":\"\",\"user_id\":0,\"user_email\":\"\",\"user_first_name\":\"\",\"user_last_name\":\"\",\"account_id\":7161,\"ip\":\"\",\"done_with_api\":false,\"platform_licences\":null},\"data\":{\"ActionID\":26412,\"ActionName\":\"Zebrocy Malware Downloader used by APT28 Threat Group .EXE File Download Variant-3\",\"AgentName\":\"VICTIM-99\",\"AssessmentName\":\"LAB02\",\"CVE\":\"_\",\"DestinationPort\":\"443\",\"File\":\"682822.exe\",\"Hash\":\"eb81c1be62f23ac7700c70d866e84f5bc354f88e6f7d84fd65374f84e252e76b\",\"Result\":{\"alert_result\":\"\",\"has_detection_result\":false,\"logging_result\":\"\",\"prevention_result\":\"blocked\"},\"RunID\":109802,\"SimulationID\":36236,\"SourcePort\":\"51967\",\"Time\":5}}" | eval json = replace(_raw, "^[^\{]+", "") ``` data emulation above ```

Hope this helps

Re: About Parsing JSON Log in splunk

huylbq — Fri, 18 Aug 2023 07:56:48 GMT

Any suggestion about Line breaker in props.conf or transform.conf

Re: About Parsing JSON Log in splunk

yuanliu — Fri, 18 Aug 2023 08:27:08 GMT

What is the problem with default line breaker? Unless you can describe a specific problem, Configure event line breaking is the best suggestion others can give.