Splunk Forwarder: Why does it stop sending data after Too Many Fields?

[ohbuckeyeio]

Hello,

We have a heavy forwarder that occasionally receives and event that exceeds the bounds of Splunk indexers. When this happens, the heavy forwarder freezes and stops sending data to the indexers. Is there a setting to tell the heavy forwarder to discard that from the queue and keep going? Our only workaround at this time is to restart the heavy forwarder.  Thank you.

This is our limits.conf:

[kv]
limit = 150
indexed_kv_limit = 0​

[richgalloway]

What is telling you why the forwarder stops sending data?  What error message(s) do you get?

What are the HF's outputs.conf settings?

[hrawat_splunk]

Here is the answer on why it's happening.
https://community.splunk.com/t5/Splunk-Enterprise/Encountered-Streaming-S2S-error-Too-many-fields/m-...

[ohbuckeyeio]

Hello Rich,

Thank you for your reply.

This is the Too Many Fields  message that begins our issue:

 07-18-2022 07:19:48.370 -0700 ERROR TcpInputProc [2721 FwdDataReceiverThread] - Encountered Streaming S2S error=Too many fields (274382) for data received from src=myhf.myco.com:62421. 

This is when the queue to the indexer shows as paused:

 07-18-2022 07:48:56.793 -0700 WARN TcpOutputProc [376 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=myindexer.myco.com inside output group my_indexers from host_src=myhf has been blocked for blocked_seconds=1740. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 

Here are the results of running btool on this particular machine:

C:\Program Files\Splunk\bin>splunk.exe btool outputs list --debug

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf [indexAndForward]

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf index = false

C:\Program Files\Splunk\etc\system\default\outputs.conf [syslog]

C:\Program Files\Splunk\etc\system\default\outputs.conf maxEventSize = 1024

C:\Program Files\Splunk\etc\system\default\outputs.conf priority = <13>

C:\Program Files\Splunk\etc\system\default\outputs.conf type = udp

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf [tcpout]

C:\Program Files\Splunk\etc\system\default\outputs.conf ackTimeoutOnShutdown = 30

C:\Program Files\Splunk\etc\system\default\outputs.conf autoLBFrequency = 30

C:\Program Files\Splunk\etc\system\default\outputs.conf autoLBVolume = 0

C:\Program Files\Splunk\etc\system\default\outputs.conf blockOnCloning = true

C:\Program Files\Splunk\etc\system\default\outputs.conf blockWarnThreshold = 100

C:\Program Files\Splunk\etc\system\default\outputs.conf cipherSuite = <removed by poster>

C:\Program Files\Splunk\etc\system\default\outputs.conf compressed = false

C:\Program Files\Splunk\etc\system\default\outputs.conf connectionTTL = 0

C:\Program Files\Splunk\etc\system\default\outputs.conf connectionTimeout = 20

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf defaultGroup = my_indexers

C:\Program Files\Splunk\etc\system\default\outputs.conf disabled = false

C:\Program Files\Splunk\etc\system\default\outputs.conf dropClonedEventsOnQueueFull = 5

C:\Program Files\Splunk\etc\system\default\outputs.conf dropEventsOnQueueFull = -1

C:\Program Files\Splunk\etc\system\default\outputs.conf ecdhCurves =<removed by poster>

C:\Program Files\Splunk\etc\system\default\outputs.conf forceTimebasedAutoLB = false

C:\Program Files\Splunk\etc\system\default\outputs.conf forwardedindex.0.whitelist = .*

C:\Program Files\Splunk\etc\system\default\outputs.conf forwardedindex.1.blacklist = _.*

C:\Program Files\Splunk\etc\system\default\outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf forwardedindex.filter.disable = true

C:\Program Files\Splunk\etc\system\default\outputs.conf heartbeatFrequency = 30

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf indexAndForward = false

C:\Program Files\Splunk\etc\system\default\outputs.conf maxConnectionsPerIndexer = 2

C:\Program Files\Splunk\etc\system\default\outputs.conf maxFailuresPerInterval = 2

C:\Program Files\Splunk\etc\system\default\outputs.conf maxQueueSize = auto

C:\Program Files\Splunk\etc\system\default\outputs.conf readTimeout = 300

C:\Program Files\Splunk\etc\system\default\outputs.conf secsInFailureInterval = 1

C:\Program Files\Splunk\etc\system\default\outputs.conf sendCookedData = true

C:\Program Files\Splunk\etc\system\default\outputs.conf sslQuietShutdown = false

C:\Program Files\Splunk\etc\system\default\outputs.conf sslVersions = tls1.2

C:\Program Files\Splunk\etc\system\default\outputs.conf tcpSendBufSz = 0

C:\Program Files\Splunk\etc\system\default\outputs.conf useACK = false

C:\Program Files\Splunk\etc\system\default\outputs.conf useClientSSLCompression = true

C:\Program Files\Splunk\etc\system\default\outputs.conf writeTimeout = 300

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf [tcpout:my_indexers]

C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf server = myindexer.myco.com:9997

[richgalloway]

A bug like this was fixed in an 8.2 maintenance release.  What version of Splunk is the HF?  if it's relatively new then consider opening a Support case on this.

Do you know the data source that is causing this error?  If so, have you checked the props.conf settings for it?

[ohbuckeyeio]

That is an interesting development. Do you happen to know the maintenance release or Issue Number where it was fixed? We are running 8.2.6

This is using the Splunk _json sourcetype which has INDEXED_EXTRACTIONS=true. We are going to make our own sourcetype with INDEXED_EXTRACTIONS=false as a workaround.

Either way, it seems there should be a way to prevent the queue from blocking even when INDEXED_EXTRACTIONS=true and the indexer field limit is hit.

[richgalloway · Accepted solution]

It was a Splunk Cloud version so perhaps the fix did not make it to an on-prem release. 

Turning off INDEXED_EXTRACTIONS = json is a good workaround and may even improve HF performance.

[ohbuckeyeio]

Thank you for the information, Rich.  I will reach out to Support.