The data flow is UFs-->HWF-->INDEXERs
Some of the event lines sizes are 100K to 300K bytes.
By default Splunk truncated the event at 10,000 bytes.
As per the props.conf, I have put the below configuration in the Indexers
props.conf [my-source-type] TRUNCATE=500000
I have restarted the indexers. But still, I see that the events are getting truncated at ~10000 characters.
Do I need to put these properties in the HWF aswell?
I have not made TRUNCATE=0 because as per the documentation, often garbage is seen when set to 0. Hence I have set this to 500000 as per the discussion with developers.
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
* Defaults to 10000 bytes.
You should set your LINEBREAKER right. That should be the first thing to check. Please post some lines on how the event starts and how it end. Try out the Preview mode in Data inputs. Check the LINEBREAKER and see if that solves it.
You have a heavy forwarder in the picture :
therefore the events are not only parsed on the indexers, but on the heavy forwarder, please put a copy of the props.conf on the HWF, and restart to apply.
Thanks for the response. But, HWF is just blindly streams out the incoming data right? It shouldn't truncate the event as it doesn't store. I'm think both LINE_BREAKER and TRUNCATE shouldn't be required at HWF. Please confirm
A heavy forwarder is an indexer with an outputs.conf. It is parsing events--it needs the LINE_BREAKER and TRUNCATE settings.
I've same problem. But I've the props.conf setting only on heavy forwarders and search heads. Do I need these settings on indexers too ?
[xml] KV_MODE = xml DATETIME_CONFIG = NONE BREAK_ONLY_BEFORE = ^\<?xml MAX_EVENTS = 500 TRUNCATE = 25000
hi @yannK ,
I already updated the props.conf in my indexer and forwarder but my data still get truncated to 100 KB.
Do you know how to find out if my data flows thru the HWF before getting into the indexer?
Hi, @mufthmu, you can look at outputs.conf on each instance to see where it's routing to. Typically, you'll need to have these line breaking rules configured on the first touch point of a full Splunk instance, whether that's a heavy forwarder or indexer.
Universal Forwarder ---> Indexers (props.conf here)
Universal Forwarder --> Heavy Forwarder(props.conf here) --> Indexers
Heavy Forwarder(props.conf here) --> Indexers
I suppose you could also install in both locations (Heavy Forwarder and Indexer) if that's simpler for you.
In the outputs.conf for your Splunk instances you'll see something like the following (often port 9997)
server=<receiving_server1>, <receiving_server2> or tcpout-server://<ipaddress_or_hostname>:<port>
if you have command line access on a Linux server you can run btool debug (your path for splunk may vary) to list out the merged configuration splunk is using for outputs.conf
/opt/splunk/bin/splunk btool --debug outputs list |egrep "server|tcpout-server" /opt/splunkforwarder/bin/splunk btool --debug outputs list |egrep "server|tcpout-server"