All Apps and Add-ons

Large JSONs are truncated at 4KB

pschildein
Explorer

TRUNCATE is set to 0 => so no truncation there, still events are truncated after 3969 signs => 4 KB on disk

Splunk runs on one system, RabbitMQ on another system, events are ingested via STDOUT

We already checked that the events are correctly transferred via RabbitMQ so there must be something in Splunk and/or the add-on. Maybe this is a problem in the Python wrapper or Java? Or are there additional settings for STDOUT?

Thanks in advance for your replies!

0 Karma

woodcock
Esteemed Legend

Here are the steps:
Put TRUNCATE=0 in props.conf for your sourcetype (if you have overridden/rewritten sourcetype, you MUST USE THE ORIGINAL VALUE).
Deploy this to all Heavy Forwarders and Indexers.
Restart all Splunk instances there.
Send new data in.
Test changes with a search using All time on the Timepicker and index_earliest=-5m in your search SPL to ensure that you are doing a valid test search.

0 Karma

Damien_Dallimor
Ultra Champion

Try TRUNCATE=0 in props.conf for your sourcetype

Many answers about this here already , just search around

0 Karma

paulbannister
Communicator

Might be unrelated, I had a similar issue with a large JSON input doing this even with Truncate set to 0

It was resolved by setting the "Response Handler" in the input to "JSONArrayHandler", this was an API input however so unsure if it relates to yours... but I thought I'd share just in case

0 Karma

pschildein
Explorer

Hi, thanks but I think that does not apply to our case. We are handling large individual json objects not arrays. Also it is not an API input but stdout.

0 Karma

Damien_Dallimor
Ultra Champion

Any errors in the logs ? (search : index=_internal error amqp.py)

What does your config look like ? (your amqp stanza in inputs.conf)

0 Karma

pschildein
Explorer

No error in the logs

[amqp://alerts]
ack_messages = 1
hec_batch_mode = 0
hec_https = 0
hostname = amqp-server
index_message_envelope = 0
index_message_propertys = 0
output_type = stdout
password = xxx
port = 5672
queue_name = amqp-queue
sourcetype = alerts
use_ssl = 0
username = xxxx
index = index
disabled = 0
exchange_name = alerts
hec_token = yyyyyyyyyyyyyyyyyyyy
host =

0 Karma

pschildein
Explorer

Hi, we still have the above mentioned problem, anything new there? Thanks! If you need more information please let me know.

0 Karma

mckenzie19
Engager

@pschildein I also have this issue. Truncating at 4KB and cannot see any logic why. Did you make any progress?

0 Karma

pschildein
Explorer

No, we changed our input to use HEC directly.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...