Getting Data In

HttpInputDataHandler parsing error: Server is busy

dilpreetsingh
Engager

Getting this errror in Heavy forwarder logs:

02-06-2020 18:28:00.283 +0000 INFO  TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
02-06-2020 18:28:08.696 +0000 ERROR HttpInputDataHandler - Parsing error : Server is busy
02-06-2020 18:28:08.703 +0000 ERROR HttpInputDataHandler - Parsing error : Server is busy
02-06-2020 18:28:09.881 +0000 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
02-06-2020 18:28:16.717 +0000 ERROR HttpInputDataHandler - Parsing error : Server is busy
02-06-2020 18:28:16.725 +0000 ERROR HttpInputDataHandler - Parsing error : Server is busy
02-06-2020 18:28:20.075 +0000 WARN  TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group US-EAST-1-INDEXER-CLUSTER has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
02-06-2020 18:28:21.717 +0000 ERROR HttpInputDataHandler - Parsing error : Server is busy
02-06-2020 18:28:21.726 +0000 ERROR HttpInputDataHandler - Parsing error : Server is busy
02-06-2020 18:28:24.881 +0000 INFO  TailReader -   ...continuing.
02-06-2020 18:28:29.847 +0000 INFO  TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.

No error spotted in indexer logs.
Data is being received via HTTP Event Collector on Heavy Forwarder.

0 Karma
1 Solution

nickhills
Ultra Champion

There are a few clues in that log.

Ther first one is this:

TailReader - Could not send data to output queue (parsingQueue)

That could indicate that your HF is recieving data faster than it can send it on to your indexers.

The next warning is therefore also important

TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group US-EAST-1-INDEXER-CLUSTER has been blocked for 10 seconds.

That means your indexers (or the network to them) is throttling your dataflow, this could be an issue on the indexers, however if you are seeing no problems on that side, then congestion or load is the likely culpret.

Now we come back to your main question:

ERROR HttpInputDataHandler - Parsing error : Server is busy

In the context of the other two messages (and the absence of problems on the indexers) suggests that you HF is struggeling to keep up, possibly network related, or possibly the rate of messages you are recieving over HEC is greater than you can parse and send for indexing.

You need to look at your queues and performance counters on the HF to establish what course of action to take next.

If my comment helps, please give it a thumbs up!

View solution in original post

nickhills
Ultra Champion

There are a few clues in that log.

Ther first one is this:

TailReader - Could not send data to output queue (parsingQueue)

That could indicate that your HF is recieving data faster than it can send it on to your indexers.

The next warning is therefore also important

TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group US-EAST-1-INDEXER-CLUSTER has been blocked for 10 seconds.

That means your indexers (or the network to them) is throttling your dataflow, this could be an issue on the indexers, however if you are seeing no problems on that side, then congestion or load is the likely culpret.

Now we come back to your main question:

ERROR HttpInputDataHandler - Parsing error : Server is busy

In the context of the other two messages (and the absence of problems on the indexers) suggests that you HF is struggeling to keep up, possibly network related, or possibly the rate of messages you are recieving over HEC is greater than you can parse and send for indexing.

You need to look at your queues and performance counters on the HF to establish what course of action to take next.

If my comment helps, please give it a thumbs up!
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...