topic Re: Splunk truncates field prior to indexing in Getting Data In

Splunk truncates field prior to indexing

bruceclarke — Tue, 07 Jul 2015 22:08:55 GMT

All,

I'm forwarding data from a Splunk forwarder that has one field with a long value (over 10k characters). I want to have Splunk index this field without truncating the value. I've set props.conf to have TRUNCATE = 0 for the appropriate sourcetype. I also modified limits.conf to have maxchars=1000000 for the kv stanza. Neither worked.

I'm also unclear if this is actually a limits issue, since I run the following query and get a different value for the length of the field. Typically the length is around 3900 characters, but it fluctuates by +/- 100 characters.

sourcetype=sourceTypeWithTruncatedField | eval l = len(truncatedField)

Why else might Splunk be truncating this field? I know the field isn't truncating in the log file we're forwarding, so I assume the issue is occurring on index.

Re: Splunk truncates field prior to indexing

MuS — Tue, 07 Jul 2015 22:29:13 GMT

Hi bruceclarke,

What kind of forwarder is it?
If it is a heavy forwarder, place the props.conf on it; if it is a universal forwarder place the props.conf on the indexer.
Read this nice wiki post to learn more about this http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

You can also check splunkd.log for something like this WARN LineBreakingProcessor - Truncating line because limit of 10000 has been exceeded to verify if it is really a truncating problem.

Also run $SPLUNK_HOME/bin/splunk cmd btool props list YourSourceType | grep TRUNCATE to verify your props.conf is applied.

Hope this helps ...

cheers, MuS

Re: Splunk truncates field prior to indexing

bruceclarke — Wed, 08 Jul 2015 18:51:34 GMT

It should be a universal forwarder, but good point. I'll double check this. And thanks for the command line options - even if they don't help debug this issue, they're great to have.

Re: Splunk truncates field prior to indexing

bruceclarke — Thu, 09 Jul 2015 14:57:38 GMT

It looks like this input was set up using a powershell script that queries a SQL database for information. I believe the truncation was actually on SQL's end. It only prints the first 8000 characters of the column.

I'm looking into the issue more, but this should be enough to go on for now.