How to get separate events from REST API input (for JSON) with historic timestamp from the json array?

[simon21]

my json looks like this:

{ [-]
_shards: { [+]
}

hits: { [-]
hits: [ [-]
{ [-]
_id: 1085_908

_index: complaint

_score: 1

_source: { [-]
addressed: 1

assigneeId: 177

assigneeName: TP_Assistant City Planner_1

categoryId: 6

categoryName: Town Planning

cityCode: 1085

cityDistrictCode: 1

cityDistrictName: SRIKAKULAM
cityDomainUrl: srikakulam.emunicipal.ap.gov.in

cityGrade: Corp

cityName: Srikakulam
cityRegionName: VISAKHAPATNAM

closed: true

closedByFunctionaryName: D S V KUMAR : Assistant City Planner

complainantEmail: abc@egov

complainantMobile: 1234567890
complainantName: ANIL KUMAR GORA

complaintAgeingFromDue: 4372

complaintAgeingdaysFromDue: 182

complaintDuration: 184

complaintGeo: { [+]
}

complaintIsClosed: Y

complaintPeriod: 184

complaintReOpenedDate: null

complaintSLADays: 48
complaintStatusName: COMPLETED

complaintTypeCode: C24

complaintTypeName: Encroachment on the public property

createdDate: 2017-05-21T12:17:38.340Z

crn: 03050-2017-AT

currentFunctionaryAgeingFromDue: 0

currentFunctionaryAssigneddate: 2017-11-21T14:52:19.571Z

currentFunctionaryIfSLA: 1

currentFunctionaryIsSLA: Y

currentFunctionaryMobileNumber: 1234567890
currentFunctionaryName: D S V KUMAR : Assistant City Planner

currentFunctionarySLADays: 48
departmentCode: TP
departmentName: TOWN PLANNING

details: despite several complaints no action from the municipal authorities

durationRange: (>30 days)
escalation1FunctionaryAgeingFromDue: 0

escalation1FunctionaryAssigneddate: null

escalation1FunctionaryIfSLA: 0

escalation1FunctionaryIsSLA: null

escalation1FunctionaryName: null

escalation1FunctionarySLADays: 0

escalation2FunctionaryAgeingFromDue: 0

escalation2FunctionaryAssigneddate: null

escalation2FunctionaryIfSLA: 0

escalation2FunctionaryIsSLA: null

escalation2FunctionaryName: null

escalation2FunctionarySLADays: 0

escalation3FunctionaryAgeingFromDue: 0

escalation3FunctionaryAssigneddate: null

escalation3FunctionaryIfSLA: 0

escalation3FunctionaryIsSLA: null

escalation3FunctionaryName: null

escalation3FunctionarySLADays: 0

escalationDate: 2017-11-22T14:52:19.438Z

escalationLevel: 3

id: 1085_908

ifClosed: 1

ifSLA: 0

inProcess: 0

initialFunctionaryAgeingFromDue: 4372

initialFunctionaryAssigneddate: 2017-05-21T12:17:38.340Z

initialFunctionaryIfSLA: 0

initialFunctionaryIsSLA: N

initialFunctionaryMobileNumber: 1234567890
initialFunctionaryName: P Kanaka Rao : Town Planning Supervisor

initialFunctionarySLADays: 48
isSLA: N

landmarkDetails:

localityGeo: { [+]
}

localityName: null

localityNo: null

reOpened: 0

reasonForRejection: null

receivingMode: MOBILE
registered: 1

rejected: 0

satisfactionIndex: 0

source: Puraseva App

url: null

wardGeo: { [+]
}

wardName: Election Ward No 36

wardNo: 36
}

_type: complaint

}

{ [-]
_id: 1016_2161

_index: complaint

_score: 1

_source: { [+]
}

_type: complaint

}

{ [+]
}

{ [+]
}

{ [+]
}

{ [+]
}

{ [+]
}

{ [+]
}

{ [+]
}

{ [+]
}

]

max_score: 1

total: 8224

}

timed_out: false

took: 5

}

Please notice I have two of "hits" header defined. The second one has the actual data. I am ingesting back data logs onto splunk, so I want to achieve two things, 1) Break the data from "_index" and 2) Have the proper timestamp parsed from the field "createdDate" for EACH EVENT. Additionally, I have tried modifying the resthandler.py but it captures the first occurrence of "hits".
I have tried indexed_extractions, it allows me to have proper timestamp by defining the timestamp_fields but the event breaking doesnt happen. I have tried kv_mode=json, but event breaking doesnt happen.
Any help would be appreciated

[Damien_Dallimor]

You should use a custom response handler.

There are many examples that ship with the app in rest_ta/bin/responsehandlers.py

Use these examples to create your own handler for your JSON format.

You then declare this custom handler to be applied in your REST stanza setup.

ie:

[manish_singh_77]

Hi,

You are breaking the event at _index and then trying to passing the created date field time to all the events. First of all, your raw data doesn't have created date field in after _ index, however if you correct your raw data, you can use the below mentioned settings in props.conf

TIME_PREFIX : createdDate:
TIME_FORMAT:%Y-%m-%dT%H:%M:%S.%3N
BREAK_ONLY_BEFORE:_index

[simon21]

This is one single json array

{ [-]
_id: 1085_908
_index: complaint
_score: 1
_source: { [-]
addressed: 1
assigneeId: 177
assigneeName: TP_Assistant City Planner_1
categoryId: 6
categoryName: Town Planning
cityCode: 1085
cityDistrictCode: 1
cityDistrictName: SRIKAKULAM
cityDomainUrl: srikakulam.emunicipal.ap.gov.in
cityGrade: Corp
cityName: Srikakulam
cityRegionName: VISAKHAPATNAM
closed: true
closedByFunctionaryName: D S V KUMAR : Assistant City Planner
complainantEmail: abc@egov
complainantMobile: 1234567890
complainantName: ANIL KUMAR GORA
complaintAgeingFromDue: 4372
complaintAgeingdaysFromDue: 182
complaintDuration: 184
complaintGeo: { [+]
}
complaintIsClosed: Y
complaintPeriod: 184
complaintReOpenedDate: null
complaintSLADays: 48
complaintStatusName: COMPLETED
complaintTypeCode: C24
complaintTypeName: Encroachment on the public property
createdDate: 2017-05-21T12:17:38.340Z
crn: 03050-2017-AT
currentFunctionaryAgeingFromDue: 0
currentFunctionaryAssigneddate: 2017-11-21T14:52:19.571Z
currentFunctionaryIfSLA: 1
currentFunctionaryIsSLA: Y
currentFunctionaryMobileNumber: 1234567890
currentFunctionaryName: D S V KUMAR : Assistant City Planner
currentFunctionarySLADays: 48
departmentCode: TP
departmentName: TOWN PLANNING
details: despite several complaints no action from the municipal authorities
durationRange: (>30 days)
escalation1FunctionaryAgeingFromDue: 0
escalation1FunctionaryAssigneddate: null
escalation1FunctionaryIfSLA: 0
escalation1FunctionaryIsSLA: null
escalation1FunctionaryName: null
escalation1FunctionarySLADays: 0
escalation2FunctionaryAgeingFromDue: 0
escalation2FunctionaryAssigneddate: null
escalation2FunctionaryIfSLA: 0
escalation2FunctionaryIsSLA: null
escalation2FunctionaryName: null
escalation2FunctionarySLADays: 0
escalation3FunctionaryAgeingFromDue: 0
escalation3FunctionaryAssigneddate: null
escalation3FunctionaryIfSLA: 0
escalation3FunctionaryIsSLA: null
escalation3FunctionaryName: null
escalation3FunctionarySLADays: 0
escalationDate: 2017-11-22T14:52:19.438Z
escalationLevel: 3
id: 1085_908
ifClosed: 1
ifSLA: 0
inProcess: 0
initialFunctionaryAgeingFromDue: 4372
initialFunctionaryAssigneddate: 2017-05-21T12:17:38.340Z
initialFunctionaryIfSLA: 0
initialFunctionaryIsSLA: N
initialFunctionaryMobileNumber: 1234567890
initialFunctionaryName: P Kanaka Rao : Town Planning Supervisor
initialFunctionarySLADays: 48
isSLA: N
landmarkDetails:
localityGeo: { [+]
}
localityName: null
localityNo: null
reOpened: 0
reasonForRejection: null
receivingMode: MOBILE
registered: 1
rejected: 0
satisfactionIndex: 0
source: Puraseva App
url: null
wardGeo: { [+]
}
wardName: Election Ward No 36
wardNo: 36
}
_type: complaint
}

this array has "CreatedDate"

It there a problem in this format of data?

[manish_singh_77]

It should work then, pls try the settings in props.conf and let me know..

[manish_singh_77]

You can use this in your search query to extract the key value pair, let me know if it works for you.

extract pairdelim=",", kvdelim='":"'

[manish_singh_77]

Have you written props.conf for this?

[simon21]

yes

props.conf

[ sourcetype_name]
BREAK_ONLY_BEFORE=_index
CHARSET=AUTO
MAX_TIMESTAMP_LOOKAHEAD=176
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
TIME_PREFIX=\"createdDate\" :
TRUNCATE=0
category=Custom
disabled=false
pulldown_type=true

Other one I tried

[ ]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\"_index\"
CHARSET=AUTO
TRUNCATE=0
disabled=false
INDEXED_EXTRACTIONS =JSON
TIMESTAMP_FIELDS =hits.hits{}._source.createdDate
LINE_BREAKER=

KV_MODE =none

[manish_singh_77]

Pls try this and let me know..

[YourSourcetypeHere]
LINE_BREAKER = hits: { [-]

[simon21]

Thanks Manish, tried it. But it didnt work.
Is there any other way to accomplish this?

[simon21]

The thing is, while i can use command at search time to split the data, but I need each event to have its proper timestamp. Which is why I need index time extractions and index time time parsing.