topic Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work in Getting Data In

LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11 — Wed, 30 Sep 2020 05:37:45 GMT

Hello Splunk TEAM,

I have a question.

I have this data:
{
"@odata.context":"https://app.inlooxnow.de/odata/$metadata#workpackageview","value":[
{
"PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","DocumentObjectRelation@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/DocumentObjectRelation"
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","DocumentObjectRelation@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/DocumentObjectRelation"
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","DocumentObjectRelation@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/DocumentObjectRelation"
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","DocumentObjectRelation@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/DocumentObjectRelation"
},{

But When I Download this data from the Rest API with JSON format and sourcetype _JSON I got all the events in one event.

I need to break this event in multiple events and extract the fields.

I try to use this:

props.conf

pulldown_type = true
LINE_BREAKER = (},{)
KV_MODE = none
category = Structured
SHOULD_LINEMERGE = false

And the data breaks correctly with (},{) but no one value is extracted to a field.

And when I try to extract data from the events I cant because never pass pass when I check regular expression and click in the event which I need to extract, after that it looking stuck.

I try to use

INDEXED_EXTRACTIONS = json

But nothing works.

Please I need a hand please!!

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

richgalloway — Thu, 04 Jun 2020 13:10:25 GMT

The LINE_BREAKER setting requires a capture group, but also discards whatever matches that capture group. In your example, the "},{" characters are thrown out so you end up with invalid json. Try LINE_BREAKER = }(,){, although that may not be much better.

Do the fields have to be extracted at index time? If not, use spath to extract them at search time.

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11 — Thu, 04 Jun 2020 13:22:42 GMT

How I use spath im so new with splunk can you explain to me please.

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

richgalloway — Thu, 04 Jun 2020 13:59:07 GMT

I have little experience with json data, but you should get some results with a simple | spath in your query.

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11 — Thu, 04 Jun 2020 14:31:17 GMT

Where I need to add this in props.conf??

Sorry for ask this 😞

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11 — Thu, 04 Jun 2020 15:18:20 GMT

Well I really don´t understand what happen I change my Line_Breaker for }(,){ and I think all work correctly. look.

Can you explain me what happen, why this only change fix this.

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11 — Thu, 04 Jun 2020 15:18:44 GMT

https://ibb.co/kh2gRrJ

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

richgalloway — Thu, 04 Jun 2020 15:22:23 GMT

LINE_BREAKER goes with the rest of your settings for this sourcetype.
spath does not go in props.conf. It's a SPL command you put in your query.

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

richgalloway — Thu, 04 Jun 2020 17:34:53 GMT

Like I said in my answer, LINE_BREAKER deletes what matches the first capture group. Throwing out curly brackets leaves invalid json, which Splunk can't process. That's why putting just a comma in the capture group works.

If your problem is resolved then please accept the answer to help future readers.

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11 — Thu, 04 Jun 2020 18:55:54 GMT

Hello,

Really Thanks for you help, you helped me fix my issue!!

but When I have all my data correct the first one event and the last continue appearing with problem.

Only two events The fist and the last.

I will show you.

https://ibb.co/zrHfJrG

and

First event.

https://ibb.co/cwV3qGD

Re: LINE_BREAKER with INDEXED_EXTRACTIONS does not work

to4kawa — Thu, 04 Jun 2020 22:02:30 GMT

your original log is valid JSON.

props.conf

TRUNCATE = 0
SHOULD_LINEMERGE = false
KV_MODE = none

that's enough.
your LINE_BREAKER makes invalid JSON event has extra ]}