Getting Data In
Highlighted

Why are larger events are truncated (10000 bytes)?

Builder

Hi,

The data flow is UFs-->HWF-->INDEXERs

Some of the event lines sizes are 100K to 300K bytes.
By default Splunk truncated the event at 10,000 bytes.
As per the props.conf, I have put the below configuration in the Indexers

props.conf
[my-source-type]
TRUNCATE=500000

I have restarted the indexers. But still, I see that the events are getting truncated at ~10000 characters.

Do I need to put these properties in the HWF aswell?

I have not made TRUNCATE=0 because as per the documentation, often garbage is seen when set to 0. Hence I have set this to 500000 as per the discussion with developers.

props.conf...
TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

Tags (2)
0 Karma
Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Motivator

You should set your LINEBREAKER right. That should be the first thing to check. Please post some lines on how the event starts and how it end. Try out the Preview mode in Data inputs. Check the LINEBREAKER and see if that solves it.

0 Karma
Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Splunk Employee
Splunk Employee

You have a heavy forwarder in the picture :
UFs-->HWF-->INDEXERs

therefore the events are not only parsed on the indexers, but on the heavy forwarder, please put a copy of the props.conf on the HWF, and restart to apply.

View solution in original post

Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Builder

Thanks for the response. But, HWF is just blindly streams out the incoming data right? It shouldn't truncate the event as it doesn't store. I'm think both LINE_BREAKER and TRUNCATE shouldn't be required at HWF. Please confirm

0 Karma
Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Splunk Employee
Splunk Employee

A heavy forwarder is an indexer with an outputs.conf. It is parsing events--it needs the LINE_BREAKER and TRUNCATE settings.

Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Builder

Thanks Yannk and sowings... It worked after placing props.conf file at Indexers and HWFs.

0 Karma
Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Engager

jayannah, Can you please send me the steps for adding props.conf to Indexers and HWF.

Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Communicator

I've same problem. But I've the props.conf setting only on heavy forwarders and search heads. Do I need these settings on indexers too ?

[xml]
KV_MODE = xml
DATETIME_CONFIG = NONE
BREAK_ONLY_BEFORE = ^\<?xml
MAX_EVENTS = 500
TRUNCATE = 25000
0 Karma
Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Explorer

hi @yannK ,
I already updated the props.conf in my indexer and forwarder but my data still get truncated to 100 KB.
Do you know how to find out if my data flows thru the HWF before getting into the indexer?

0 Karma
Highlighted

Re: Why are larger events are truncated (10000 bytes)?

Motivator

Hi, @mufthmu, you can look at outputs.conf on each instance to see where it's routing to. Typically, you'll need to have these line breaking rules configured on the first touch point of a full Splunk instance, whether that's a heavy forwarder or indexer.

i.e.
Universal Forwarder ---> Indexers (props.conf here)
OR
Universal Forwarder --> Heavy Forwarder(props.conf here) --> Indexers
OR
Heavy Forwarder(props.conf here) --> Indexers

I suppose you could also install in both locations (Heavy Forwarder and Indexer) if that's simpler for you.

In the outputs.conf for your Splunk instances you'll see something like the following (often port 9997)

server=<receiving_server1>, <receiving_server2>
or tcpout-server://<ipaddress_or_hostname>:<port>

if you have command line access on a Linux server you can run btool debug (your path for splunk may vary) to list out the merged configuration splunk is using for outputs.conf

example:

  /opt/splunk/bin/splunk btool --debug outputs list |egrep "server|tcpout-server"
  /opt/splunkforwarder/bin/splunk btool --debug outputs list |egrep "server|tcpout-server"
0 Karma