I open a new thread because in the previous one I was reviewing several errors at the same time
for this specific error message I have already read all the forum posts and the ones I have found on the internet but I am still having problems
• I have no Licence problems
• I don't have another output.conf file in the heavy forwarder
• I do not have a high performance or consumption in IOPS after consulting with the command iostats and iotop -
The devices through syslog send the logs to the heavy carrier that receives them and when reviewing them the logs are with the date and time updated all the time
For some reason the heavy forwarder doesn't constantly forward the logs to the indexers, as when querying with
index = * host = xxxx | statistics count per host _time
I see records but with delays of 8-10 hours.
When checking the logs in var / log / splunk / splunkd.log
I use grep xxxx splunkd.log to check the errors only from the host that interests me and that is where I see the message
09-21-2020 07:20:48.483 -0500 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=indexer inside output group default-autolb -group from host_src=xxxxx has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receivin g system's health in the Splunk Monitoring Console. It is probably not accepting data.
the heavy forwarder does not have high IOPS problems
The problem seems to be in the indexers that are not able to receive the information.
In that order of ideas, what do you recommend reviewing?
The servers configured in outputs.conf are not performing well. there could be many reasons:
check all servers (indexers) in outputs.conf of Heavy forwarder are healthy (CPU and memory utilization).
check if you have deployed outputs.conf to indexers by mistake. generally indexers don't have outputs.conf.