Break single line of data.

[BB2]

I have a question about breaking up a single line of data to send to the Splunk Indexer. 

We sending data which can have over 50,000 characters on a single line.  I would like to know if there is a way to break up the data on the source server with the universal forwarder before sending it to the indexer and then reassemble it after it arrives at the indexer.  

We would like to know if this is possible rather than having to increase the Truncate size on the indexer to take all the data at once.  

[BB2 · Accepted solution]

Thanks for all the info.

We are going to go with increasing the truncate on the index server.

[PickleRick]

It's a bit more complicated than that. Data is not sent from UF as events (unless you're using indexed extractions), it's getting sent as chunks (which can cause issues if you have big events and don't have event breaker configured properly).

And it's actually the other way around - you want to have event breaker (not line breaker! since no line breaking happens on UF) set so that events are _not_ split between two different chunks. The cause for it is that two chunks of data can go to different outputs from the same group and end up on two different indexers.

So even if there was a way to reassemble an event you wouldn't have anything to reassemble it from.

Long story short - you want to make sure your events are _not_ getting split.

[gcusello]

Hi @BB2 ,

only one question: why?

if the issue is the limit of 50,000 chars, you can only increase the TRUNCATE limit.

There's no utility (even if it's possible but not!) to trucate an event on forwarders and then reassemble it  on Indexers because events are compressed and stored in packets and sent from Forwarders to Indexers with no relation with the lenght of the event.

So I ask you again why?

the only action that you must do is increasing the lenght of the events aging on the TRUNCATE parameters.

Ciao.

Giuseppe

[sainag_splunk]

You can apply EVENT BREAKER  settings on your props.conf. 

• Go to your app/local directory on your Deployment server.
• Create or edit props.conf file. 
• Update the EVENT_BREAKER with the appropriate regex pattern for your source. Typically, this is the same as your LINE_BREAKER regex.
• Reload the serverclass app on the Deployment server.
• Verify that the updated props.conf is successfully deployed to the Universal Forwarder.
• That should complete the setup.

Refer: https://community.splunk.com/t5/Getting-Data-In/How-to-apply-EVENT-BREAKER-on-UF-for-better-data-dis...


Hope this helps.