Large JSONs are truncated at 4KB

[pschildein]

TRUNCATE is set to 0 => so no truncation there, still events are truncated after 3969 signs => 4 KB on disk

Splunk runs on one system, RabbitMQ on another system, events are ingested via STDOUT

We already checked that the events are correctly transferred via RabbitMQ so there must be something in Splunk and/or the add-on. Maybe this is a problem in the Python wrapper or Java? Or are there additional settings for STDOUT?

Thanks in advance for your replies!

[woodcock]

Here are the steps:
Put TRUNCATE=0 in props.conf for your sourcetype (if you have overridden/rewritten sourcetype, you MUST USE THE ORIGINAL VALUE).
Deploy this to all Heavy Forwarders and Indexers.
Restart all Splunk instances there.
Send new data in.
Test changes with a search using All time on the Timepicker and index_earliest=-5m in your search SPL to ensure that you are doing a valid test search.

[Damien_Dallimor]

Try TRUNCATE=0 in props.conf for your sourcetype

Many answers about this here already , just search around

[paulbannister]

Might be unrelated, I had a similar issue with a large JSON input doing this even with Truncate set to 0

It was resolved by setting the "Response Handler" in the input to "JSONArrayHandler", this was an API input however so unsure if it relates to yours... but I thought I'd share just in case

[pschildein]

Hi, thanks but I think that does not apply to our case. We are handling large individual json objects not arrays. Also it is not an API input but stdout.

[Damien_Dallimor]

Any errors in the logs ? (search : index=_internal error amqp.py)

What does your config look like ? (your amqp stanza in inputs.conf)

[pschildein]

No error in the logs

[amqp://alerts]
ack_messages = 1
hec_batch_mode = 0
hec_https = 0
hostname = amqp-server
index_message_envelope = 0
index_message_propertys = 0
output_type = stdout
password = xxx
port = 5672
queue_name = amqp-queue
sourcetype = alerts
use_ssl = 0
username = xxxx
index = index
disabled = 0
exchange_name = alerts
hec_token = yyyyyyyyyyyyyyyyyyyy
host =

[pschildein]

Hi, we still have the above mentioned problem, anything new there? Thanks! If you need more information please let me know.

[mckenzie19]

@pschildein I also have this issue. Truncating at 4KB and cannot see any logic why. Did you make any progress?

[pschildein]

No, we changed our input to use HEC directly.