topic Re: Break single line of data. in Getting Data In

Break single line of data.

BB2 — Fri, 18 Oct 2024 20:17:13 GMT

I have a question about breaking up a single line of data to send to the Splunk Indexer.

We sending data which can have over 50,000 characters on a single line. I would like to know if there is a way to break up the data on the source server with the universal forwarder before sending it to the indexer and then reassemble it after it arrives at the indexer.

We would like to know if this is possible rather than having to increase the Truncate size on the indexer to take all the data at once.

Re: Break single line of data.

sainag_splunk — Fri, 18 Oct 2024 22:56:37 GMT

You can apply EVENT BREAKER settings on your props.conf.

Go to your app/local directory on your Deployment server.
Create or edit props.conf file.
Update the EVENT_BREAKER with the appropriate regex pattern for your source. Typically, this is the same as your LINE_BREAKER regex.
Reload the serverclass app on the Deployment server.
Verify that the updated props.conf is successfully deployed to the Universal Forwarder.
That should complete the setup.

Refer: https://community.splunk.com/t5/Getting-Data-In/How-to-apply-EVENT-BREAKER-on-UF-for-better-data-distribution/m-p/614423

Hope this helps.

Re: Break single line of data.

gcusello — Sat, 19 Oct 2024 09:17:28 GMT

Hi @BB2 ,

only one question: why?

if the issue is the limit of 50,000 chars, you can only increase the TRUNCATE limit.

There's no utility (even if it's possible but not!) to trucate an event on forwarders and then reassemble it on Indexers because events are compressed and stored in packets and sent from Forwarders to Indexers with no relation with the lenght of the event.

So I ask you again why?

the only action that you must do is increasing the lenght of the events aging on the TRUNCATE parameters.

Ciao.

Giuseppe

Re: Break single line of data.

PickleRick — Sat, 19 Oct 2024 21:15:04 GMT

It's a bit more complicated than that. Data is not sent from UF as events (unless you're using indexed extractions), it's getting sent as chunks (which can cause issues if you have big events and don't have event breaker configured properly).

And it's actually the other way around - you want to have event breaker (not line breaker! since no line breaking happens on UF) set so that events are _not_ split between two different chunks. The cause for it is that two chunks of data can go to different outputs from the same group and end up on two different indexers.

So even if there was a way to reassemble an event you wouldn't have anything to reassemble it from.

Long story short - you want to make sure your events are _not_ getting split.

Re: Break single line of data.

BB2 — Mon, 21 Oct 2024 16:33:22 GMT

Thanks for all the info.

We are going to go with increasing the truncate on the index server.