Getting Data In

LINE_BREAKER with INDEXED_EXTRACTIONS does not work

jaciro11
Path Finder

Hello Splunk TEAM,

I have a question.

I have this data:
{
"@odata.context":"https://app.inlooxnow.de/odata/$metadata#workpackageview","value":[
{
"PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{
PlanningReservationId":"5345345","DoneDate":null,"WorkAmount":261.0,"IsDone":false,"Name":"Informaci","StartReminderDateTime":null,"EndReminderDateTime":null,"ProjectId":"4d7a-8fb1-b69918c35e25","ContactId":null,"ChangedDate":"2017-10-24T13:51:03.277Z","CreatedDate":"2017-09-04T14:16:31.147Z","PositionNumber":0,"CardPositionNumber":0,"PositionOrderedByContact":null,"PlanningId":null,"GroupId":"76879d8a-482b-b17d-4d0cb8eee218","CreatedByContactId":"b-85a6-4af8-a7b2-ca8dc38cd601","WorkPackageStartDateTime":null,"WorkPackageEndDateTime":"2017-10-17T23:00:00Z","IsBillable":true,"IsBilled":false,"CostPerHour":null,"SecondaryCostPerHour":null,"CustomColor":null,"IsRead":true,"AssignedByContactId":null,"HasStartDate":false,"HasEndDate":true,"StartDateTime":null,"EndDateTime":"2017-10-17T23:00:00Z","PlanningReservationStatusName":"To do","PlanningReservationStatusId":"263d4762-dc54-46ec-95a7-36391f9ef4b5","MindMapNodeId":null,"PlanningTypeId":null,"Location":null,"PSPCode":null,"ColorFlag":null,"PlanningCustomColor":null,"PlanningProgress":null,"CalendarId":null,"ConstraintType":null,"ConstraintDate":null,"DurationTicks":null,"PlanningStartDateTime":null,"PlanningEndDateTime":null,"IsFixed":null,"IsMilestone":null,"IsGrouping":null,"IsCollapsed":null,"IsCritical":null,"PlanningSnapshotId":null,"OriginalPlanningId":null,"IsProjectGrouping":null,"IsVIP":null,"PlanningPositionNumber":null,"DisplayName":null,"ParentPlanningId":null,"ProjectName":"Hospital","ProjectNumber":"1257","Priority":1,"FirstManager":"pepe raul","FirstTeamMember":"Inloox - Heremias","FirstCustomer":null,"FirstPartner":null,"FirstAdditionalContact":null,"ComputedProgress":100.0,"IsEndDateFixed":false,"IsRecycled":false,"IsArchived":true,"IsRequest":false,"ProjectImageId":"2b797791-ccb5-490b-9939-2dfda829f3f4","ClientName":"hospitalo","ClientNumber":"0114","ProjectStatusName":"Completed","ProjectStatusProgress":100,"DivisionName":null,"ContactName":null,"FirstName":null,"LastName":null,"ContactDisplayName":null,"ContactImageId":null,"AssignedByDisplayName":null,"AssignedByImageId":null,"GroupName":"Preventa","WorkpackagePredecessorDone":null,"AllWorkpackagePredecessorDone":null,"CustomExpand@odata.navigationLink":"https://app.inlooxnow.de/odata/workpackageview(50c9fca1-b316-4b96-abb6-0018d1a58c61)/CustomExpand","..."
},{

But When I Download this data from the Rest API with JSON format and sourcetype _JSON I got all the events in one event.

I need to break this event in multiple events and extract the fields.

I try to use this:

props.conf

pulldown_type = true
LINE_BREAKER = (},{)
KV_MODE = none
category = Structured
SHOULD_LINEMERGE = false

And the data breaks correctly with (},{) but no one value is extracted to a field.
alt text

And when I try to extract data from the events I cant because never pass pass when I check regular expression and click in the event which I need to extract, after that it looking stuck.
alt text

I try to use

INDEXED_EXTRACTIONS = json

But nothing works.

Please I need a hand please!!

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting requires a capture group, but also discards whatever matches that capture group. In your example, the "},{" characters are thrown out so you end up with invalid json. Try LINE_BREAKER = }(,){, although that may not be much better.

Do the fields have to be extracted at index time? If not, use spath to extract them at search time.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

to4kawa
Ultra Champion

your original log is valid JSON.

props.conf

TRUNCATE = 0
SHOULD_LINEMERGE = false
KV_MODE = none

that's enough.
your LINE_BREAKER makes invalid JSON event has extra ]}

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The LINE_BREAKER setting requires a capture group, but also discards whatever matches that capture group. In your example, the "},{" characters are thrown out so you end up with invalid json. Try LINE_BREAKER = }(,){, although that may not be much better.

Do the fields have to be extracted at index time? If not, use spath to extract them at search time.

---
If this reply helps you, an upvote would be appreciated.

jaciro11
Path Finder

Hello,

Really Thanks for you help, you helped me fix my issue!!

but When I have all my data correct the first one event and the last continue appearing with problem.

Only two events The fist and the last.

I will show you.

https://ibb.co/zrHfJrG

and

First event.

https://ibb.co/cwV3qGD

0 Karma

jaciro11
Path Finder

Well I really don´t understand what happen I change my Line_Breaker for }(,){ and I think all work correctly. look.

alt text

Can you explain me what happen, why this only change fix this.

0 Karma

jaciro11
Path Finder
0 Karma

jaciro11
Path Finder

How I use spath im so new with splunk can you explain to me please.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I have little experience with json data, but you should get some results with a simple | spath in your query.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

jaciro11
Path Finder

Where I need to add this in props.conf??

Sorry for ask this 😞

0 Karma

richgalloway
SplunkTrust
SplunkTrust

LINE_BREAKER goes with the rest of your settings for this sourcetype.
spath does not go in props.conf. It's a SPL command you put in your query.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Like I said in my answer, LINE_BREAKER deletes what matches the first capture group. Throwing out curly brackets leaves invalid json, which Splunk can't process. That's why putting just a comma in the capture group works.

If your problem is resolved then please accept the answer to help future readers.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...