topic Re: Splunk Cloud HTTP-INPUT Truncating to 10000 Characters in Getting Data In

How to set Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

CS_ — Wed, 16 Feb 2022 15:20:51 GMT

Hi all,

We are using Splunk Cloud, and I am using the https://http-inputs-mydomain.com/services/collector/raw to send a log file for ingestion.

The problem is that each line in this log file can be quite big, 25000 characters or more.

Splunk Cloud is truncating at 10,000 characters.

I can find steps for handling this on Splunk On-Prem for Heavy Log Forwarders etc. but doesn't seem to be addressed for the http-inputs on cloud.

Any idea's on how I can change it to accept larger logs?

Thanks,

Chris

Re: Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

tshah-splunk — Tue, 15 Feb 2022 11:56:41 GMT

Hey @CS_,

Can you try defining the TRUNCATE parameter in the props.conf file for the sourcetype that you have defined in the inputs.conf for HEC Token input. The default value of this parameter is 10000, hence the reason for truncating at 10000 characters. You can increase the value to accept larger logs.

More details for the parameter can be found here - https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Propsconf

Re: Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

CS_ — Tue, 15 Feb 2022 12:03:27 GMT

thanks @tshah-splunk I'll have a look and see if I can make that change. will update with result.

-C

Re: Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

VatsalJagani — Wed, 16 Feb 2022 04:46:26 GMT

Yes, the TRUNCATE parameter should definitely work as you are using services/collector/raw endpoint for HEC.

Just to keep in mind, the TRUNCATE parameter or any other index-time/parsing parameters will not work if you are using just the regular /services/collector endpoint.

Re: How to set Splunk Cloud HTTP-INPUT Truncating to 10000 Characters

CS_ — Wed, 16 Feb 2022 16:35:26 GMT

We were able to get it working by amending the source type on the HEC token, and setting the character limit to 1mb (1,000,000 characters)