How to split json array into multiple events?

[Sanjai676]

Hi ,

I have this json data which I am unable to parse through any of the props.conf mechanisms.

{"meta": {"limit": 20, "next": "/api/v1/indicators/?username=xxxx&api_key=xxxxxxxxxxxxx&limit=20&offset=20", "offset": 0, "previous": null, "total_count": 570289}, "objects": [{"_id": "570f4f34c011fb78b52434d7", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "BR", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:08.677000", "impact": {"analyst": "Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:08.680000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "Basic_Feed", "date": "2016-04-14 04:05:08.679000", "method": "basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T19:26:46Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "177.33.224.193"}, {"_id": "570f4f62c011fb78b52435a1", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "US", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:54.227000", "impact": {"analyst": "_Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:54.229000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "_Basic_Feed", "date": "2016-04-14 04:05:54.229000", "method": "_basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T21:26:52Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "104.238.191.144"}, 

The log is json like format, although events appear to be in one single line and I'm unable to break them using line breakers.

This is how my props.conf looks like after several different tries:

[sourcetype = _json]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = (\{|\[\s+{)
MUST_BREAK_AFTER = (\}|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_trailing_commas = s/\},/}/g
SEDCMD-remove_footer = s/\]\s+\}//g
TIME_PREFIX = \"modified\":\s+\"

Please help.

[Sanjai676 · Accepted solution]

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

[alexwade13]

Hey! im having a similar issues, theres an array in my json that i want to grab and separate out as separate events. BREAK_ONLY_AFTER has been giving me some difficulties, what exactly do you mean by using a custom json array handler? I've gotten the data in via the REST API, all i need to do is parse it correctly.
EDIT:
I've found a different way, using SEDCMD to get rid of headers and footers of the object, and LINEBREAKER starting at the beginning of each event. i had an issue where LINEBREAKER wasn't working, where it was taking away everything in my parens, but i solved that by giving it just the comma inside the parens to eat, followed by the previous regex i had

[ryanoconnor]

See if you can start by validating the JSON. The following website is a good resource for that:

http://jsonlint.com/

Once you've done this, the easiest way to break up your JSON into key/value format would be using props.conf and setting KV_MODE to JSON. http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

[Sanjai676]

i have checked and validated the json data. I had tried the KV_MODE setting,but didn't workout. I found a thread which is very similar to what i'm facing.
https://answers.splunk.com/answers/289520/how-to-split-a-json-array-into-multiple-events-wit.html
Although the same logic isn't working in my case.