Getting Data In

Can I change TRUNCATE and MAX_EVENTS to unlimited ?

mataharry
Communicator

I have large logs, with more than 10000 chars per line, and multiline events as large as whole XML file.
They are always truncated or cut in multiple events because too long.

It seems that the default is

MAX_EVENTS=250
TRUNCATE=10000

Can I push them to unlimited (i.e. 0)?

Tags (1)
1 Solution

yannK
Splunk Employee
Splunk Employee

First of all, the TRUNCATE and MAX_EVENTS exists to avoid memory leak, indexing slowdown and search slowdown. Please never to use unlimited values (TRUNCATE=0 or MAX_EVENTS=0), they can potentially cause a nasty event to blow up your Splunk.

That said :

  • TRUNCATE defines the number of characters per line, once reached exceed characters are dropped.
  • MAX_EVENTS defines the maximum number of lines per multi-line events. Once reached the event is broken, and exceeding lines are interpreted as a new events (sometimes causing a new timestamp detection)

You can increase those values increased accordingly to your needs for specific sourcetypes.
A good method to monitor the result is to use searches like :
sourcetype=mysourcetype | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

View solution in original post

Tejkumar451
Explorer

its been 4 years, and yes you can do that but it not preferable 🙂

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Try using the following in your props.conf file for the sourcetype you want to do this to:

SEDCMD-truncate = s/\(.\{1,10000\}\).*/\\1/

I could be wrong on some of the backslashes for the Splunk SEDCMD, and it may not work exactly with the multi-line events that you have, but in actual sed, this will truncate lines (remember, sed is a line-based stream editor) to a max of 10000 characters. I don't have any of your data to see if it will work either, so your mileage may vary. At least it is a place to start.

theunf
Communicator

An how about the second part after props: the transform part ?

There i´m using SEDCMD or a transformation that is truncating very large messages.
Tried lookahead = 1024000 but it´s still trunkating ...

How can i set this to unlimited ?

yannK
Splunk Employee
Splunk Employee

First of all, the TRUNCATE and MAX_EVENTS exists to avoid memory leak, indexing slowdown and search slowdown. Please never to use unlimited values (TRUNCATE=0 or MAX_EVENTS=0), they can potentially cause a nasty event to blow up your Splunk.

That said :

  • TRUNCATE defines the number of characters per line, once reached exceed characters are dropped.
  • MAX_EVENTS defines the maximum number of lines per multi-line events. Once reached the event is broken, and exceeding lines are interpreted as a new events (sometimes causing a new timestamp detection)

You can increase those values increased accordingly to your needs for specific sourcetypes.
A good method to monitor the result is to use searches like :
sourcetype=mysourcetype | eval length=len(_raw) | stats max(length) perc95(length) max(linecount) perc95(linecount)

sandeepreddy947
Path Finder

I had about 2 million characters in a single line, As I can't use TRUNCATE=0. What should I do in this case?
Is there any way I can get all those 2million characters into Splunk.
Currently my props are:
[source::truncate_value/source/with_large_character_in_single_line]
TRUNCATE=100000

0 Karma

Roy_9
Motivator

Hello @sandeepreddy947 ,

How did you overcome your problem? I have a similar issue where particular events are more than 1 million bytes.

 

Thanks

0 Karma

mufthmu
Path Finder

hi sandeep, did u get an answer to this?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...