There are multiple json messages in _raw into a HEC and splitting them doesn't quite get what I need?

[loganseth]

Greetings.

We recently turned on a HEC and have JSON data coming in and I have noticed that multiple JSON blobs are embedded in _raw.  I searched several solutions and found one that actually did parse _raw into a new colum "split_raw" and then I went so far as to try

| eval raw=split_raw

but when I do

| table *

it still shows all the data from the first entry only

I think my questions are:

1. the ones that are 'multiple json entries' I think is when 'a bunch arrive at about the same time' - so is there a way to FORCE these to split at ingestion (to guarantee 1:1 json-to-event)? my guess is i may need to play with my source_type, but looking for some guidance/thoughts.

2. if not, will have to split them (like the link above) and then do some processing on the new split_raw field?

Thank you so much for leads and thoughts on this!

[yuanliu · Accepted solution]

Split mangled data at search time should be your last resort.  If you only recently started this HEC, better start over.

Line breaking (more correctly, the document is about event breaking) is the foundation of data ingestion; it is highly tunable.   Your source developer should make every effort to make sure Splunk can break events easily.  For example, does the source insert a new line ("\n") after each JSON event?  Is there some marker to mark the beginning and end of an event?

[PickleRick]

It's also worth noting that if you submit your events to /event endpoint, you skip some pipeline steps completely (most notably event breaking and - by default - timestamp parsing).

[loganseth]

thank you for this perspective. that's how i felt, as well, that

Split mangled data at search time should be your last resort.

i did some research on LINE_BREAK and went back to the default \n\s one I found (I had changed it) and premlinary results are going great!!

---