Getting Data In

Help with props.conf to break events

vrmandadi
Builder

I am trying to work on props.conf to parse and break correctly.I am pushing data using CURL commands but it is sending 50 logs in one event.It worked through UI but failing when sent from CURL commands.I want to break it into individual events .Only the first event start with   "{"sourcetype": "json","event": {"

AND ends with "last_updated" (EXAMPLE:"last_updated": "2022-03-24T02:35:41.148727Z"
},) .Rest of the events START WITH ID and end with last_updated....There are lot of nested ID in the event which I did not post but the syntax should be something that will break after last_updated

 

I want the events to BREAK AFTER THE "last_updated"  followed by closed flower brackets and the new event should start from 

NOTE:ONLY THE first event start is different ..rest all events start with id and end with last_updated.

 

I tried BREAK_ONLY_BEFORE=\"\w*\"\:\s\"\d*\-\d*\-\d*\w\d*\:\d*\:\d*\.\d*\w\" ... but its not breaking correctly

{
"id": 

 

Following are the sample events that I want to break

Event1:

 

{"sourcetype": "json","event": {
.
.
.
.
.
},
"created": "2022-02-07",
"last_updated": "2022-03-24T02:35:41.083145Z"

 

Event 2:

 

{
"id": 150749,
"name": "no hostname 1660322000234",
.
.
.
.
.
"created": "2022-02-07",
"last_updated": "2022-03-24T02:35:41.148727Z"
}

 

I used the below props...it worked uploading sample file via GUI but when I used this sourcetype in CURL through HEC it is not breaking.

[ Netbox ]
CHARSET=UTF-8
DATETIME_CONFIG=CURRENT
LINE_BREAKER=([\r\n]+)\s+{
MUST_BREAK_BEFORE=\"\w*\"\:\s\"\d*\-\d*\-\d*\w\d*\:\d*\:\d*\.\d*\w\"
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
category=Custom
disabled=false
pulldown_type=true

 

CURL:

curl -k http://10.xx.xx.xx:8088/services/collector/event -H 'Authorization: Splunk <TOKEN>' -d '{"sourcetype": "Netbox","event": '"$SITEINFO"'}'

 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

It doesn't work that way. The HEC /collector/event endpoint is meant for already split events. On ingestion from this endpoint Splunk skips the event breaking altogether because you already said where the event starts and where it ends. You should split it before and send as multiple events. You can however send multiple events in one HTTP transaction. Just make sure they are properly formed. Like

{ "sourcetype":"whatever","index":"test1","event":"a b c d"}
{"event":"another one"}
{"index":"test2","event":{"field1":"value1","field2":"value2"}}

I'm not sure if you need commas between the events.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...