Getting Data In

How to set Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

CS_
Path Finder

Hi all,

We are using Splunk Cloud, and I am using the https://http-inputs-mydomain.com/services/collector/raw to send a log file for ingestion.

The problem is that each line in this log file can be quite big, 25000 characters or more.

Splunk Cloud is truncating at 10,000 characters.

I can find steps for handling this on Splunk On-Prem for Heavy Log Forwarders etc. but doesn't seem to be addressed for the http-inputs on cloud.

Any idea's on how I can change it to accept larger logs?

Thanks,

Chris

Labels (1)
0 Karma
1 Solution

tshah-splunk
Splunk Employee
Splunk Employee

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf 

---
If you find the answer helpful, an upvote/karma is appreciated

View solution in original post

CS_
Path Finder

We were able to get it working by amending the source type on the HEC token, and setting the character limit to 1mb (1,000,000 characters)

0 Karma

tshah-splunk
Splunk Employee
Splunk Employee

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf 

---
If you find the answer helpful, an upvote/karma is appreciated

CS_
Path Finder

thanks @tshah-splunk I'll have a look and see if I can make that change. will update with result.

-C

VatsalJagani
SplunkTrust
SplunkTrust

Yes, the TRUNCATE parameter should definitely work as you are using services/collector/raw endpoint for HEC.

Just to keep in mind, the TRUNCATE parameter or any other index-time/parsing parameters will not work if you are using just the regular /services/collector endpoint.

Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...