topic Re: Logs truncated in Splunk despite line being under the 10000 bytes threshold in Getting Data In

Logs truncated in Splunk despite line being under the 10000 bytes threshold

victorcorrea — Wed, 23 Oct 2024 15:34:40 GMT

Hi community,

I have observed an issue with the ingestion of the first line in a log file that, at first glance, seemed to have been truncated. Here's a screenshot for reference:

My apologies for the poor job at blurring the data, but the first event should look like the second event, with a whole lot of data after the highlighted field.

The field DistPoint itself should have a value of "DEPSY.IM2" and, it got, apparently, truncated at such a weird point.

All other subsequent lines in the log were successfully ingested.

There were 3 log files landing on the ingestion point in quick succession - seconds apart, so I am not sure if this could have been the issue.

I was about to update the truncate value for the sourcetype, but all lines in the logs are 3551 bytes, by default.

Any ideas as to what could the problem have been?

Thank you.

Re: Logs truncated in Splunk despite line being under the 10000 bytes threshold

yuanliu — Thu, 24 Oct 2024 05:42:47 GMT

Most likely there's some line breaking problem. Documentation is Configure event line breaking (and the entire Configure event processing. You would also get better discussion in the forum Getting Data In.

Re: Logs truncated in Splunk despite line being under the 10000 bytes threshold

PickleRick — Thu, 24 Oct 2024 12:47:47 GMT

It might also be the issue with badly/not set EVENT_BREAKER (which is not the same as LINE_BREAKER).

Moving the discussion to Getting Data In.

Re: Logs truncated in Splunk despite line being under the 10000 bytes threshold

victorcorrea — Wed, 30 Oct 2024 18:37:27 GMT

Looks like the issue was with "LINE_MERGE=TRUE" in the props.conf file.

Thank you @PickleRick and @yuanliu for chiming in.