Getting Data In

LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11
Path Finder

Hello Splunk TEAM,

I have a question.

I have this data:
{
"@odata.context":"https://app.inlooxnow.de/odata/$metadata#workpackageview","value":[
{
"PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{

But When I Download this data from the Rest API with JSON format and sourcetype _JSON I got all the events in one event.

I need to break this event in multiple events and extract the fields.

I try to use this:

props.conf

pulldown_type = true
LINE_BREAKER = (},{)
KV_MODE = none
category = Structured
SHOULD_LINEMERGE = false

And the data breaks correctly with (},{) but no one value is extracted to a field.
alt text

And when I try to extract data from the events I cant because never pass pass when I check regular expression and click in the event which I need to extract, after that it looking stuck.
alt text

I try to use

INDEXED_EXTRACTIONS = json

But nothing works.

Please I need a hand please!!

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting requires a capture group, but also discards whatever matches that capture group. In your example, the "},{" characters are thrown out so you end up with invalid json. Try LINE_BREAKER = }(,){, although that may not be much better.

Do the fields have to be extracted at index time? If not, use spath to extract them at search time.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

to4kawa
Ultra Champion

your original log is valid JSON.

props.conf

TRUNCATE = 0
SHOULD_LINEMERGE = false
KV_MODE = none

that's enough.
your LINE_BREAKER makes invalid JSON event has extra ]}

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting requires a capture group, but also discards whatever matches that capture group. In your example, the "},{" characters are thrown out so you end up with invalid json. Try LINE_BREAKER = }(,){, although that may not be much better.

Do the fields have to be extracted at index time? If not, use spath to extract them at search time.

---
If this reply helps you, Karma would be appreciated.

jaciro11
Path Finder

Hello,

Really Thanks for you help, you helped me fix my issue!!

but When I have all my data correct the first one event and the last continue appearing with problem.

Only two events The fist and the last.

I will show you.

https://ibb.co/zrHfJrG

and

First event.

https://ibb.co/cwV3qGD

0 Karma

jaciro11
Path Finder

Well I really don´t understand what happen I change my Line_Breaker for }(,){ and I think all work correctly. look.

alt text

Can you explain me what happen, why this only change fix this.

0 Karma

jaciro11
Path Finder
0 Karma

jaciro11
Path Finder

How I use spath im so new with splunk can you explain to me please.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I have little experience with json data, but you should get some results with a simple | spath in your query.

---
If this reply helps you, Karma would be appreciated.
0 Karma

jaciro11
Path Finder

Where I need to add this in props.conf??

Sorry for ask this 😞

0 Karma

richgalloway
SplunkTrust
SplunkTrust

LINE_BREAKER goes with the rest of your settings for this sourcetype.
spath does not go in props.conf. It's a SPL command you put in your query.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Like I said in my answer, LINE_BREAKER deletes what matches the first capture group. Throwing out curly brackets leaves invalid json, which Splunk can't process. That's why putting just a comma in the capture group works.

If your problem is resolved then please accept the answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...