topic Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ? in Getting Data In

Can I change TRUNCATE and MAX_EVENTS to unlimited ?

mataharry — Thu, 06 Jun 2013 00:23:52 GMT

I have large logs, with more than 10000 chars per line, and multiline events as large as whole XML file.
They are always truncated or cut in multiple events because too long.

It seems that the default is

MAX_EVENTS=250
TRUNCATE=10000

Can I push them to unlimited (i.e. 0)?

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

yannK — Thu, 06 Jun 2013 00:25:32 GMT

First of all, the TRUNCATE and MAX_EVENTS exists to avoid memory leak, indexing slowdown and search slowdown. Please never to use unlimited values (TRUNCATE=0 or MAX_EVENTS=0), they can potentially cause a nasty event to blow up your Splunk.

That said :

TRUNCATE defines the number of characters per line, once reached exceed characters are dropped.
MAX_EVENTS defines the maximum number of lines per multi-line events. Once reached the event is broken, and exceeding lines are interpreted as a new events (sometimes causing a new timestamp detection)

You can increase those values increased accordingly to your needs for specific sourcetypes.
A good method to monitor the result is to use searches like :
sourcetype=mysourcetype | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

theunf — Fri, 19 Sep 2014 12:59:18 GMT

An how about the second part after props: the transform part ?

There i´m using SEDCMD or a transformation that is truncating very large messages.
Tried lookahead = 1024000 but it´s still trunkating ...

How can i set this to unlimited ?

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

cpetterborg — Thu, 20 Nov 2014 16:41:05 GMT

Try using the following in your props.conf file for the sourcetype you want to do this to:

SEDCMD-truncate = s/\(.\{1,10000\}\).*/\\1/

I could be wrong on some of the backslashes for the Splunk SEDCMD, and it may not work exactly with the multi-line events that you have, but in actual sed, this will truncate lines (remember, sed is a line-based stream editor) to a max of 10000 characters. I don't have any of your data to see if it will work either, so your mileage may vary. At least it is a place to start.

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

Tejkumar451 — Thu, 11 May 2017 19:13:14 GMT

its been 4 years, and yes you can do that but it not preferable 🙂

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

sandeepreddy947 — Tue, 29 Sep 2020 23:50:15 GMT

I had about 2 million characters in a single line, As I can't use TRUNCATE=0. What should I do in this case?
Is there any way I can get all those 2million characters into Splunk.
Currently my props are:
[source::truncate_value/source/with_large_character_in_single_line]
TRUNCATE=100000

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

mufthmu — Thu, 12 Dec 2019 21:19:06 GMT

hi sandeep, did u get an answer to this?

Re: Can I change TRUNCATE and MAX_EVENTS to unlimited ?

Roy_9 — Tue, 07 Dec 2021 01:56:45 GMT

Hello @sandeepreddy947 ,

How did you overcome your problem? I have a similar issue where particular events are more than 1 million bytes.

Thanks