Getting Data In

There are multiple json messages in _raw into a HEC and splitting them doesn't quite get what I need?

loganseth
Path Finder

Greetings.

We recently turned on a HEC and have JSON data coming in and I have noticed that multiple JSON blobs are embedded in _raw.  I searched several solutions and found one that actually did parse _raw into a new colum "split_raw" and then I went so far as to try

 

 

| eval raw=split_raw

 

 

but when I do

 

 

| table *

 

 

it still shows all the data from the first entry only

I think my questions are:

1. the ones that are 'multiple json entries' I think is when 'a bunch arrive at about the same time' - so is there a way to FORCE these to split at ingestion (to guarantee 1:1 json-to-event)? my guess is i may need to play with my source_type, but looking for some guidance/thoughts.

2. if not, will have to split them (like the link above) and then do some processing on the new split_raw field?

Thank you so much for leads and thoughts on this!

Labels (2)
0 Karma
1 Solution

yuanliu
SplunkTrust
SplunkTrust

Split mangled data at search time should be your last resort.  If you only recently started this HEC, better start over.

Line breaking (more correctly, the document is about event breaking) is the foundation of data ingestion; it is highly tunable.   Your source developer should make every effort to make sure Splunk can break events easily.  For example, does the source insert a new line ("\n") after each JSON event?  Is there some marker to mark the beginning and end of an event?

View solution in original post

yuanliu
SplunkTrust
SplunkTrust

Split mangled data at search time should be your last resort.  If you only recently started this HEC, better start over.

Line breaking (more correctly, the document is about event breaking) is the foundation of data ingestion; it is highly tunable.   Your source developer should make every effort to make sure Splunk can break events easily.  For example, does the source insert a new line ("\n") after each JSON event?  Is there some marker to mark the beginning and end of an event?

PickleRick
SplunkTrust
SplunkTrust

It's also worth noting that if you submit your events to /event endpoint, you skip some pipeline steps completely (most notably event breaking and - by default - timestamp parsing).

0 Karma

loganseth
Path Finder

thank you for this perspective. that's how i felt, as well, that

Split mangled data at search time should be your last resort.

i did some research on LINE_BREAK and went back to the default \n\s one I found (I had changed it) and premlinary results are going great!!


Tags (1)
0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...