Getting Data In
Highlighted

How to split json array into multiple events?

Path Finder

Hi ,

I have this json data which I am unable to parse through any of the props.conf mechanisms.

{"meta": {"limit": 20, "next": "/api/v1/indicators/?username=xxxx&api_key=xxxxxxxxxxxxx&limit=20&offset=20", "offset": 0, "previous": null, "total_count": 570289}, "objects": [{"_id": "570f4f34c011fb78b52434d7", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "BR", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:08.677000", "impact": {"analyst": "Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:08.680000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "Basic_Feed", "date": "2016-04-14 04:05:08.679000", "method": "basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T19:26:46Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "177.33.224.193"}, {"_id": "570f4f62c011fb78b52435a1", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "US", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:54.227000", "impact": {"analyst": "_Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:54.229000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "_Basic_Feed", "date": "2016-04-14 04:05:54.229000", "method": "_basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T21:26:52Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "104.238.191.144"}, 

The log is json like format, although events appear to be in one single line and I'm unable to break them using line breakers.

This is how my props.conf looks like after several different tries:

[sourcetype = _json]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = (\{|\[\s+{)
MUST_BREAK_AFTER = (\}|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_trailing_commas = s/\},/}/g
SEDCMD-remove_footer = s/\]\s+\}//g
TIME_PREFIX = \"modified\":\s+\"

Please help.

0 Karma
Highlighted

Re: How to split json array into multiple events?

Builder

See if you can start by validating the JSON. The following website is a good resource for that:

http://jsonlint.com/

Once you've done this, the easiest way to break up your JSON into key/value format would be using props.conf and setting KV_MODE to JSON. http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

0 Karma
Highlighted

Re: How to split json array into multiple events?

Path Finder

i have checked and validated the json data. I had tried the KV_MODE setting,but didn't workout. I found a thread which is very similar to what i'm facing.
https://answers.splunk.com/answers/289520/how-to-split-a-json-array-into-multiple-events-wit.html
Although the same logic isn't working in my case.

0 Karma
Highlighted

Re: How to split json array into multiple events?

Path Finder

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

View solution in original post

0 Karma
Highlighted

Re: How to split json array into multiple events?

Engager

Hey! im having a similar issues, theres an array in my json that i want to grab and separate out as separate events. BREAKONLYAFTER has been giving me some difficulties, what exactly do you mean by using a custom json array handler? I've gotten the data in via the REST API, all i need to do is parse it correctly.
EDIT:
I've found a different way, using SEDCMD to get rid of headers and footers of the object, and LINEBREAKER starting at the beginning of each event. i had an issue where LINEBREAKER wasn't working, where it was taking away everything in my parens, but i solved that by giving it just the comma inside the parens to eat, followed by the previous regex i had

0 Karma