How to configure Unviversal forwarder Props.conf for Truncate

[splunkreal]

If we index JSON data from HF and want to extend TRUNCATE setting for long logs, should we also do it on (clustered) indexers or HF is enough?

Also when looking at https://community.splunk.com/t5/Splunk-Search/Why-are-not-all-field-values-are-extracted-for-long-JS... it seems TRUNCATE=0 isn't enough?

Thanks!

[ddrillic]

TRUNCATE has its default value at 10,000 characters but a JSON event is a one-liner, and trimming it "breaks" the entire presentation of the object, so we internally here decided to make the default unlimited for JSON cases, the only thing is to find a way to classify them automatically and not handle each JSON case individually. If anybody has a generic solution, please share.

[PickleRick]

Trimming json structure prematurely causes it to not be a well-formed json structure. So it doesn't break "presentation" but makes it unparseable.

[ddrillic]

Sorry @PickleRick, but try please to view a truncated json object, it's a mess ; -) 

[PickleRick]

Yes. But it's because it stops being a well-formed json object and therefore Splunk cannot parse it as a json object and therefore cannot show you its structure. It just shows you raw event contents.

So we're both kinda right. But the underlying cause is not the presentation itself but the inability to parse and interpret the data as json object.

[PickleRick]

As a rule of thumb (with some small exceptions) index-time settings "work" on the first "heavy" component your event goes through. So if you have an inermediate HF layer so that events from your UFs go to HFs and from them are forwarded to indexers, you need those settingns on the HFs. If you have a standalone HF ingesting jsons (from HEC input, pulling from remote with API and so on) you also need those settings on this HF.

[richgalloway]

The TRUNCATE setting applies to the first full instance of Splunk (HF or indexer) that the data passes through.  TRUNCATE=0 should accept events of any length.  What makes you think it isn't "enough"?

[splunkreal]

Thanks @richgalloway for being so fast, so it means HF then.

It seems for long json logs there is another setting to put.

[PickleRick]

The TRUNCATE setting is responsible for truncating overly long events. It has nothing to do (except of course for truncating the json prematurely) with any other problems with parsing json/extracting fields from it.