Getting Data In

How to set Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

CS_
Path Finder

Hi all,

We are using Splunk Cloud, and I am using the https://http-inputs-mydomain.com/services/collector/raw to send a log file for ingestion.

The problem is that each line in this log file can be quite big, 25000 characters or more.

Splunk Cloud is truncating at 10,000 characters.

I can find steps for handling this on Splunk On-Prem for Heavy Log Forwarders etc. but doesn't seem to be addressed for the http-inputs on cloud.

Any idea's on how I can change it to accept larger logs?

Thanks,

Chris

Labels (1)
0 Karma
1 Solution

tshah-splunk
Splunk Employee
Splunk Employee

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf 

---
If you find the answer helpful, an upvote/karma is appreciated

View solution in original post

CS_
Path Finder

We were able to get it working by amending the source type on the HEC token, and setting the character limit to 1mb (1,000,000 characters)

0 Karma

tshah-splunk
Splunk Employee
Splunk Employee

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf 

---
If you find the answer helpful, an upvote/karma is appreciated

CS_
Path Finder

thanks @tshah-splunk I'll have a look and see if I can make that change. will update with result.

-C

VatsalJagani
SplunkTrust
SplunkTrust

Yes, the TRUNCATE parameter should definitely work as you are using services/collector/raw endpoint for HEC.

Just to keep in mind, the TRUNCATE parameter or any other index-time/parsing parameters will not work if you are using just the regular /services/collector endpoint.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...