Getting Data In

How to split json array into multiple events?

Sanjai676
Path Finder

Hi ,

I have this json data which I am unable to parse through any of the props.conf mechanisms.

{"meta": {"limit": 20, "next": "/api/v1/indicators/?username=xxxx&api_key=xxxxxxxxxxxxx&limit=20&offset=20", "offset": 0, "previous": null, "total_count": 570289}, "objects": [{"_id": "570f4f34c011fb78b52434d7", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "BR", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:08.677000", "impact": {"analyst": "Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:08.680000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "Basic_Feed", "date": "2016-04-14 04:05:08.679000", "method": "basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T19:26:46Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "177.33.224.193"}, {"_id": "570f4f62c011fb78b52435a1", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "US", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:54.227000", "impact": {"analyst": "_Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:54.229000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "_Basic_Feed", "date": "2016-04-14 04:05:54.229000", "method": "_basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T21:26:52Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "104.238.191.144"}, 

The log is json like format, although events appear to be in one single line and I'm unable to break them using line breakers.

This is how my props.conf looks like after several different tries:

[sourcetype = _json]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = (\{|\[\s+{)
MUST_BREAK_AFTER = (\}|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_trailing_commas = s/\},/}/g
SEDCMD-remove_footer = s/\]\s+\}//g
TIME_PREFIX = \"modified\":\s+\"

Please help.

0 Karma
1 Solution

Sanjai676
Path Finder

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

View solution in original post

0 Karma

Sanjai676
Path Finder

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

0 Karma

alexwade13
Engager

Hey! im having a similar issues, theres an array in my json that i want to grab and separate out as separate events. BREAK_ONLY_AFTER has been giving me some difficulties, what exactly do you mean by using a custom json array handler? I've gotten the data in via the REST API, all i need to do is parse it correctly.
EDIT:
I've found a different way, using SEDCMD to get rid of headers and footers of the object, and LINEBREAKER starting at the beginning of each event. i had an issue where LINEBREAKER wasn't working, where it was taking away everything in my parens, but i solved that by giving it just the comma inside the parens to eat, followed by the previous regex i had

0 Karma

ryanoconnor
Builder

See if you can start by validating the JSON. The following website is a good resource for that:

http://jsonlint.com/

Once you've done this, the easiest way to break up your JSON into key/value format would be using props.conf and setting KV_MODE to JSON. http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

0 Karma

Sanjai676
Path Finder

i have checked and validated the json data. I had tried the KV_MODE setting,but didn't workout. I found a thread which is very similar to what i'm facing.
https://answers.splunk.com/answers/289520/how-to-split-a-json-array-into-multiple-events-wit.html
Although the same logic isn't working in my case.

0 Karma
Get Updates on the Splunk Community!

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...

4 Ways the Splunk Community Helps You Prepare for .conf25

.conf25 is right around the corner, and whether you’re a first-time attendee or a seasoned Splunker, the ...

Enhance Your Splunk App Development: New Tools & Support

UCC FrameworkAdd-on Builder has been around for quite some time. It helps build Splunk apps faster, but it ...