Getting Data In

Splunk Forwarder: Why does it stop sending data after Too Many Fields?

ohbuckeyeio
Communicator

Hello,

We have a heavy forwarder that occasionally receives and event that exceeds the bounds of Splunk indexers. When this happens, the heavy forwarder freezes and stops sending data to the indexers. Is there a setting to tell the heavy forwarder to discard that from the queue and keep going? Our only workaround at this time is to restart the heavy forwarder.  Thank you.

This is our limits.conf:

 

[kv]
limit = 150
indexed_kv_limit = 0​

 

 

Labels (2)
Tags (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

It was a Splunk Cloud version so perhaps the fix did not make it to an on-prem release. 

Turning off INDEXED_EXTRACTIONS = json is a good workaround and may even improve HF performance.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

What is telling you why the forwarder stops sending data?  What error message(s) do you get?

What are the HF's outputs.conf settings?

---
If this reply helps you, Karma would be appreciated.
0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

ohbuckeyeio
Communicator
Hello Rich,

Thank you for your reply.

This is the Too Many Fields  message that begins our issue:

 07-18-2022 07:19:48.370 -0700 ERROR TcpInputProc [2721 FwdDataReceiverThread] - Encountered Streaming S2S error=Too many fields (274382) for data received from src=myhf.myco.com:62421. 

This is when the queue to the indexer shows as paused:

 07-18-2022 07:48:56.793 -0700 WARN TcpOutputProc [376 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=myindexer.myco.com inside output group my_indexers from host_src=myhf has been blocked for blocked_seconds=1740. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 

Here are the results of running btool on this particular machine:

C:\Program Files\Splunk\bin>splunk.exe btool outputs list --debug
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf [indexAndForward]
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf index = false
C:\Program Files\Splunk\etc\system\default\outputs.conf [syslog]
C:\Program Files\Splunk\etc\system\default\outputs.conf maxEventSize = 1024
C:\Program Files\Splunk\etc\system\default\outputs.conf priority = <13>
C:\Program Files\Splunk\etc\system\default\outputs.conf type = udp
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf [tcpout]
C:\Program Files\Splunk\etc\system\default\outputs.conf ackTimeoutOnShutdown = 30
C:\Program Files\Splunk\etc\system\default\outputs.conf autoLBFrequency = 30
C:\Program Files\Splunk\etc\system\default\outputs.conf autoLBVolume = 0
C:\Program Files\Splunk\etc\system\default\outputs.conf blockOnCloning = true
C:\Program Files\Splunk\etc\system\default\outputs.conf blockWarnThreshold = 100
C:\Program Files\Splunk\etc\system\default\outputs.conf cipherSuite = <removed by poster>
C:\Program Files\Splunk\etc\system\default\outputs.conf compressed = false
C:\Program Files\Splunk\etc\system\default\outputs.conf connectionTTL = 0
C:\Program Files\Splunk\etc\system\default\outputs.conf connectionTimeout = 20
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf defaultGroup = my_indexers
C:\Program Files\Splunk\etc\system\default\outputs.conf disabled = false
C:\Program Files\Splunk\etc\system\default\outputs.conf dropClonedEventsOnQueueFull = 5
C:\Program Files\Splunk\etc\system\default\outputs.conf dropEventsOnQueueFull = -1
C:\Program Files\Splunk\etc\system\default\outputs.conf ecdhCurves =<removed by poster>
C:\Program Files\Splunk\etc\system\default\outputs.conf forceTimebasedAutoLB = false
C:\Program Files\Splunk\etc\system\default\outputs.conf forwardedindex.0.whitelist = .*
C:\Program Files\Splunk\etc\system\default\outputs.conf forwardedindex.1.blacklist = _.*
C:\Program Files\Splunk\etc\system\default\outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf forwardedindex.filter.disable = true
C:\Program Files\Splunk\etc\system\default\outputs.conf heartbeatFrequency = 30
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf indexAndForward = false
C:\Program Files\Splunk\etc\system\default\outputs.conf maxConnectionsPerIndexer = 2
C:\Program Files\Splunk\etc\system\default\outputs.conf maxFailuresPerInterval = 2
C:\Program Files\Splunk\etc\system\default\outputs.conf maxQueueSize = auto
C:\Program Files\Splunk\etc\system\default\outputs.conf readTimeout = 300
C:\Program Files\Splunk\etc\system\default\outputs.conf secsInFailureInterval = 1
C:\Program Files\Splunk\etc\system\default\outputs.conf sendCookedData = true
C:\Program Files\Splunk\etc\system\default\outputs.conf sslQuietShutdown = false
C:\Program Files\Splunk\etc\system\default\outputs.conf sslVersions = tls1.2
C:\Program Files\Splunk\etc\system\default\outputs.conf tcpSendBufSz = 0
C:\Program Files\Splunk\etc\system\default\outputs.conf useACK = false
C:\Program Files\Splunk\etc\system\default\outputs.conf useClientSSLCompression = true
C:\Program Files\Splunk\etc\system\default\outputs.conf writeTimeout = 300
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf [tcpout:my_indexers]
C:\Program Files\Splunk\etc\apps\MY_outputs\default\outputs.conf server = myindexer.myco.com:9997
0 Karma

richgalloway
SplunkTrust
SplunkTrust

A bug like this was fixed in an 8.2 maintenance release.  What version of Splunk is the HF?  if it's relatively new then consider opening a Support case on this.

Do you know the data source that is causing this error?  If so, have you checked the props.conf settings for it?

---
If this reply helps you, Karma would be appreciated.
0 Karma

ohbuckeyeio
Communicator

That is an interesting development. Do you happen to know the maintenance release or Issue Number where it was fixed? We are running 8.2.6

This is using the Splunk _json sourcetype which has INDEXED_EXTRACTIONS=true. We are going to make our own sourcetype with INDEXED_EXTRACTIONS=false as a workaround.

Either way, it seems there should be a way to prevent the queue from blocking even when INDEXED_EXTRACTIONS=true and the indexer field limit is hit.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It was a Splunk Cloud version so perhaps the fix did not make it to an on-prem release. 

Turning off INDEXED_EXTRACTIONS = json is a good workaround and may even improve HF performance.

---
If this reply helps you, Karma would be appreciated.

ohbuckeyeio
Communicator

Thank you for the information, Rich.  I will reach out to Support.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...