Getting Data In

How to extract required data from the HEC _raw event and truncate the remaining information before indexing

New Member

My single event length is too long so I want to extract and ingest the specific part from it. The part is in the middle of the event, so I tried extracting it using BREAK_ONLY_BEFORE and BREAK_ONLY_AFTER. Also used the LINE_BREAKER function but it is not working as expected. How can we define start and end of the log in the props.conf file? Is there any alternative to achieve this?

Log sample:


0 Karma

Ultra Champion

BREAK_ONLY_BEFORE and BREAK_ONLY_AFTER are settings for when you have SHOULD_LINEMERGE set (which you should not have set unless there is really no other way).

To cut data from the event before indexing you'd normally use SEDCMD but there is one caveat - you must of course write proper regex. And with json it's kinda hard. You can't simply get anything from one brace to another because:

1) You can have nested entities. But even if you can assume that you won't have any "substructures" in your json,

2) A closing brace might be simply contained within a string.

You can use ingest-time eval to extract a given path from a json structure using json_extract() but that's "heavier" solution than simple regex-based operations.

So it's not that easy. The right approach will depend on the actual data you have and how much you can safely assume about its structure.


0 Karma
Get Updates on the Splunk Community!

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...

New Customer Testimonials

Enterprises of all sizes and across different industries are accelerating cloud adoption by migrating ...