topic Re: How to split json array into multiple events? in Getting Data In

How to split json array into multiple events?

Sanjai676 — Wed, 25 May 2016 11:07:44 GMT

Hi ,

I have this json data which I am unable to parse through any of the props.conf mechanisms.

{"meta": {"limit": 20, "next": "/api/v1/indicators/?username=xxxx&api_key=xxxxxxxxxxxxx&limit=20&offset=20", "offset": 0, "previous": null, "total_count": 570289}, "objects": [{"_id": "570f4f34c011fb78b52434d7", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "BR", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:08.677000", "impact": {"analyst": "Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:08.680000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "Basic_Feed", "date": "2016-04-14 04:05:08.679000", "method": "basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T19:26:46Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "177.33.224.193"}, {"_id": "570f4f62c011fb78b52435a1", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "US", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:54.227000", "impact": {"analyst": "_Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:54.229000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "_Basic_Feed", "date": "2016-04-14 04:05:54.229000", "method": "_basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T21:26:52Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "104.238.191.144"},

The log is json like format, although events appear to be in one single line and I'm unable to break them using line breakers.

This is how my props.conf looks like after several different tries:

[sourcetype = _json]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = (\{|\[\s+{)
MUST_BREAK_AFTER = (\}|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_trailing_commas = s/\},/}/g
SEDCMD-remove_footer = s/\]\s+\}//g
TIME_PREFIX = \"modified\":\s+\"

Please help.

Re: How to split json array into multiple events?

ryanoconnor — Wed, 25 May 2016 13:45:39 GMT

See if you can start by validating the JSON. The following website is a good resource for that:

http://jsonlint.com/

Once you've done this, the easiest way to break up your JSON into key/value format would be using props.conf and setting KV_MODE to JSON. http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

Re: How to split json array into multiple events?

Sanjai676 — Thu, 26 May 2016 07:45:34 GMT

i have checked and validated the json data. I had tried the KV_MODE setting,but didn't workout. I found a thread which is very similar to what i'm facing.
https://answers.splunk.com/answers/289520/how-to-split-a-json-array-into-multiple-events-wit.html
Although the same logic isn't working in my case.

Re: How to split json array into multiple events?

Sanjai676 — Fri, 27 May 2016 09:23:06 GMT

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

Re: How to split json array into multiple events?

alexwade13 — Tue, 29 Sep 2020 15:31:53 GMT

Hey! im having a similar issues, theres an array in my json that i want to grab and separate out as separate events. BREAK_ONLY_AFTER has been giving me some difficulties, what exactly do you mean by using a custom json array handler? I've gotten the data in via the REST API, all i need to do is parse it correctly.
EDIT:
I've found a different way, using SEDCMD to get rid of headers and footers of the object, and LINEBREAKER starting at the beginning of each event. i had an issue where LINEBREAKER wasn't working, where it was taking away everything in my parens, but i solved that by giving it just the comma inside the parens to eat, followed by the previous regex i had