Getting Data In

How to set Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

CS_
Path Finder

Hi all,

We are using Splunk Cloud, and I am using the https://http-inputs-mydomain.com/services/collector/raw to send a log file for ingestion.

The problem is that each line in this log file can be quite big, 25000 characters or more.

Splunk Cloud is truncating at 10,000 characters.

I can find steps for handling this on Splunk On-Prem for Heavy Log Forwarders etc. but doesn't seem to be addressed for the http-inputs on cloud.

Any idea's on how I can change it to accept larger logs?

Thanks,

Chris

Labels (1)
0 Karma
1 Solution

tshah-splunk
Splunk Employee
Splunk Employee

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf 

---
If you find the answer helpful, an upvote/karma is appreciated

View solution in original post

CS_
Path Finder

We were able to get it working by amending the source type on the HEC token, and setting the character limit to 1mb (1,000,000 characters)

0 Karma

tshah-splunk
Splunk Employee
Splunk Employee

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf 

---
If you find the answer helpful, an upvote/karma is appreciated

CS_
Path Finder

thanks @tshah-splunk I'll have a look and see if I can make that change. will update with result.

-C

VatsalJagani
Champion

Yes, the TRUNCATE parameter should definitely work as you are using services/collector/raw endpoint for HEC.

Just to keep in mind, the TRUNCATE parameter or any other index-time/parsing parameters will not work if you are using just the regular /services/collector endpoint.

Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...