Can I change TRUNCATE and MAX_EVENTS to unlimited ?

[mataharry]

I have large logs, with more than 10000 chars per line, and multiline events as large as whole XML file.
They are always truncated or cut in multiple events because too long.

It seems that the default is

MAX_EVENTS=250
TRUNCATE=10000

Can I push them to unlimited (i.e. 0)?

[Tejkumar451]

its been 4 years, and yes you can do that but it not preferable 🙂

[cpetterborg]

Try using the following in your props.conf file for the sourcetype you want to do this to:

SEDCMD-truncate = s/\(.\{1,10000\}\).*/\\1/

I could be wrong on some of the backslashes for the Splunk SEDCMD, and it may not work exactly with the multi-line events that you have, but in actual sed, this will truncate lines (remember, sed is a line-based stream editor) to a max of 10000 characters. I don't have any of your data to see if it will work either, so your mileage may vary. At least it is a place to start.

[theunf]

An how about the second part after props: the transform part ?

There i´m using SEDCMD or a transformation that is truncating very large messages.
Tried lookahead = 1024000 but it´s still trunkating ...

How can i set this to unlimited ?

[yannK · Accepted solution]

First of all, the TRUNCATE and MAX_EVENTS exists to avoid memory leak, indexing slowdown and search slowdown. Please never to use unlimited values (TRUNCATE=0 or MAX_EVENTS=0), they can potentially cause a nasty event to blow up your Splunk.

That said :

• TRUNCATE defines the number of characters per line, once reached exceed characters are dropped.
• MAX_EVENTS defines the maximum number of lines per multi-line events. Once reached the event is broken, and exceeding lines are interpreted as a new events (sometimes causing a new timestamp detection)

You can increase those values increased accordingly to your needs for specific sourcetypes.
A good method to monitor the result is to use searches like :
sourcetype=mysourcetype | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

[sandeepreddy947]

I had about 2 million characters in a single line, As I can't use TRUNCATE=0. What should I do in this case?
Is there any way I can get all those 2million characters into Splunk.
Currently my props are:
[source::truncate_value/source/with_large_character_in_single_line]
TRUNCATE=100000

[Roy_9]

Hello @sandeepreddy947 ,

How did you overcome your problem? I have a similar issue where particular events are more than 1 million bytes.

Thanks

[mufthmu]

hi sandeep, did u get an answer to this?