Getting Data In

Help with props.conf to break events

vrmandadi
Builder

I am trying to work on props.conf to parse and break correctly.I am pushing data using CURL commands but it is sending 50 logs in one event.It worked through UI but failing when sent from CURL commands.I want to break it into individual events .Only the first event start with   "{"sourcetype": "json","event": {"

AND ends with "last_updated" (EXAMPLE:"last_updated": "2022-03-24T02:35:41.148727Z"
},) .Rest of the events START WITH ID and end with last_updated....There are lot of nested ID in the event which I did not post but the syntax should be something that will break after last_updated

 

I want the events to BREAK AFTER THE "last_updated"  followed by closed flower brackets and the new event should start from 

NOTE:ONLY THE first event start is different ..rest all events start with id and end with last_updated.

 

I tried BREAK_ONLY_BEFORE=\"\w*\"\:\s\"\d*\-\d*\-\d*\w\d*\:\d*\:\d*\.\d*\w\" ... but its not breaking correctly

{
"id": 

 

Following are the sample events that I want to break

Event1:

 

{"sourcetype": "json","event": {
.
.
.
.
.
},
"created": "2022-02-07",
"last_updated": "2022-03-24T02:35:41.083145Z"

 

Event 2:

 

{
"id": 150749,
"name": "no hostname 1660322000234",
.
.
.
.
.
"created": "2022-02-07",
"last_updated": "2022-03-24T02:35:41.148727Z"
}

 

I used the below props...it worked uploading sample file via GUI but when I used this sourcetype in CURL through HEC it is not breaking.

[ Netbox ]
CHARSET=UTF-8
DATETIME_CONFIG=CURRENT
LINE_BREAKER=([\r\n]+)\s+{
MUST_BREAK_BEFORE=\"\w*\"\:\s\"\d*\-\d*\-\d*\w\d*\:\d*\:\d*\.\d*\w\"
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
category=Custom
disabled=false
pulldown_type=true

 

CURL:

curl -k http://10.xx.xx.xx:8088/services/collector/event -H 'Authorization: Splunk <TOKEN>' -d '{"sourcetype": "Netbox","event": '"$SITEINFO"'}'

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

It doesn't work that way. The HEC /collector/event endpoint is meant for already split events. On ingestion from this endpoint Splunk skips the event breaking altogether because you already said where the event starts and where it ends. You should split it before and send as multiple events. You can however send multiple events in one HTTP transaction. Just make sure they are properly formed. Like

{ "sourcetype":"whatever","index":"test1","event":"a b c d"}
{"event":"another one"}
{"index":"test2","event":{"field1":"value1","field2":"value2"}}

I'm not sure if you need commas between the events.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...