topic Re: how to break the json data into events? in Getting Data In

how to break the json data into events?

chaseto — Fri, 20 Nov 2015 16:53:31 GMT

Hello experts,below is the json data

{"actions": [{"date": "2012-05-17 00:00:00", "action": "Read for the first time and referred to the Senate committee on Judiciary", "+amendment": null, "type": ["bill:reading:1", "committee:referred"], "actor": "upper"}, {"date": "2012-05-18 00:00:00", "action": "Read for the second time and placed on the calendar", "+amendment": null, "type": ["bill:reading:2"], "actor": "upper"}, {"date": "2012-05-21 00:00:00", "action": "Unfinished Business", "+amendment": null, "type": ["other"], "actor": "upper"}..................................

I want to break the above data starting with a date into single events,Is that possible

I tried using sourcetype=json and kv_mode=json but still it is not breaking .Please help

Re: how to break the json data into events?

chaseto — Fri, 20 Nov 2015 18:32:33 GMT

i want date,action,type,bill,commite,actor in one event

Re: how to break the json data into events?

jeffland — Mon, 23 Nov 2015 17:50:08 GMT

You can use this in props.conf to break your data into events starting with your date field:

LINE_BREAKER = (..){"date"

You should try it out during the "Add Data" wizard to see if everything is as you want it to be.

Re: how to break the json data into events?

chaseto — Mon, 23 Nov 2015 22:05:34 GMT

Hey Jeff thanks for the response

(..) is this also needed to be added

Re: how to break the json data into events?

chaseto — Mon, 23 Nov 2015 22:11:52 GMT

I tried doing that...In the add data..set source type...but it is not showing only as a single event

Re: how to break the json data into events?

jeffland — Mon, 30 Nov 2015 12:10:30 GMT

(..) will tell splunk to take away the two characters before the opening curly bracket {. In your data, that is either a space and an opening bracket [, or a comma and a space. From props.conf:

* The regex must contain a capturing group -- a pair of parentheses which defines an identified subcomponent of the match.
* Wherever the regex matches, Splunk considers the start of the first capturing group to be the end of the previous event, and considers the end of the first capturing group to be the start of the next event.
* The contents of the first capturing group are discarded, and will not be present in any event.  You are telling Splunk that this text comes between lines.

These settings work for me, but I've encountered some strange behavior with the "Add Data" wizard from time to time. I'd suggest you set up and save your settings for the source type from the source type manager (Settings - Source Types) and select that saved source type during the "Add Data" wizard, that should make everything apply correctly even during that wizard.

For completeness' sake, here are the settings you should definitely have for your sourcetype:

LINE_BREAKER = (..){"date"
SHOULD_LINEMERGE = false

Re: how to break the json data into events?

chaseto — Mon, 30 Nov 2015 18:35:34 GMT

well I tried using your settings but the whole data is coming as one single event instead of breaking into multiple events.

Re: how to break the json data into events?

jeffland — Tue, 01 Dec 2015 06:50:34 GMT

That's strange. If I paste your sample from above in a text document and try to index it with the above settings, they work. I'm on 6.3.1, but I doubt version will make a difference here.

Maybe there are other characters in the original data?

Re: how to break the json data into events?

chaseto — Tue, 01 Dec 2015 19:59:02 GMT

Name Value

CHARSET=UTF-8

INDEXED_EXTRACTIONS=json

KV_MODE=JSON

SHOULD_LINEMERGE=false

category=Structured

description=JavaScript Object Notation format. For more information, visit http://json.org/

disabled=false

pulldown_type=true

LINE_BREAKER= (..){"date"

the above is the setting I have done,is that fine or any changes needed

Re: how to break the json data into events?

jeffland — Tue, 29 Sep 2020 08:01:46 GMT

Almost. You don't need KV_MODE when you set INDEXED_EXTRACTIONS - the first one is for search time field extraction, which you don't need because you already have indexed extractions. Therefore, you should set KV_MODE = none.

Re: how to break the json data into events?

chaseto — Wed, 02 Dec 2015 17:32:24 GMT

I have tried that too,but still the event summary still shows only one event

Event Summary

Sample size 10000 bytes
Events 1
Event time distribution
0
1
5/17/12 12:00 AM: 1
5/17/12 12:00 AM5/17/12 12:00 AM
Number of lines per event

of lines # of events % of events

1 1

Re: how to break the json data into events?

jeffland — Thu, 03 Dec 2015 08:14:06 GMT

Hm, I just noticed you have a space before the line breaker regex - if you remove that, restart your splunk and try to add the data once more, I don't know why it shouldn't work. It does on my machine.

Re: how to break the json data into events?

vrmandadi — Thu, 03 Mar 2016 22:21:36 GMT

in splunk web when you onboard the file you will have option for selecting the sourcetype...i think in web structured section you have _json...if you select that particular type splunk automatically breaks the events