How to get large a JSON file recognized as JSON in Splunk Web and prevent it from being truncated?

[tleyden]

We're pushing a few different JSON files to our Splunk server via a Splunk Forwarder running on a different machine. With the smaller JSON file (https://gist.github.com/tleyden/d6d29fd5442c512405b6) (11k), it seems to be understood as JSON by the server UI, and it allows tree-style navigation:

OTOH, with a bigger JSON file (https://gist.github.com/7f562131e54239250318) (65k), it does not have the same tree-style navigation, and it appears to be being truncated (and I guess the two are probably related)

How can I fix this? Where is the truncation happening? On the machine running the forwarder, or the server where it's being forwarded to? (I'm guessing the former, and so the configuration fix would have to be done on the forwarder)

The way the JSON is being forwarded is by this forwarding rule:

sudo /opt/splunkforwarder/bin/splunk add monitor /tmp/jsontest -index main -sourcetype sync_gateway_expvars

and there has been no additional configuration for our custom sourcetype (and in fact, I later realized that this sourcetype is probably being misused, and the source should be sync_gateway_expvars)

I'm a splunk n00b, so please don't assume very much knowledge. Thanks in advance for any help!

[ltrand]

1: fixing truncate:
in props.conf do the following:

[sourcetype_name]
TRUNCATE = 0

If it is truncating, then it could be loosing some of the JSON formatting and therefore breaking the JSON element extraction, I don't doubt that they are related.

[tleyden]

I'm having trouble figuring out which props.conf to modify. I've tried modifying both etc/apps/Splunk_TA_nix/local/props.conf and etc/system/local/props.conf on the machine were the splunk forwarder is running, and neither seem to have any effect -- the JSON is still truncated when it's shown on the UI. I did restart the splunk forwarder process after changing the props.conf files. Is there any way to debug why values in props.conf are ignored?

[ltrand]

You'll want to look for the props.conf inside of any app that is picking up the source /home/centos/debug_vars.log. That app should be configured on everything from the forwarder to the final indexer where the data will sit. If you only do the forwarder, then the indexer, not knowing any better, will then use its default setting for ingestion.

Which app props.conf you change doesn't really matter, with the exception of "don't edit anything in default". It's just there to help us manage precidence and make it easier for us humans. The system will rollup all of the props into a single running config at start. I generally make sure that end-to-end data handling is done within a single "app" so that it's easiest to troubleshoot.

If you looked into your splunk logs, you'll probably see a bunch of truncation messages, though the logging is not the most verbose or easy to use. I have not found a way to easily debug props/transforms/inputs issues other than getting very familiar with reading Splunk's internal logs.