Why is syslogs Forwarder set up not working and paused data flow?

[bbbb21]

I have set up the Universal Forwarder locally in my machine using this guide

https://splunk.paloaltonetworks.com/universal-forwarder.html

/opt/splunkforwarder/etc/system/local/inputs.conf

[monitor:///var/log/udp514.log]
sourcetype = pan:log
disabled =0

/opt/splunkforwarder/etc/system/local/outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = andrea-xps-15-7590:9997
disabled=false
[tcpout-server://andrea-xps-15-7590:9997]

(the local ip becomes 'andrea-xps-15-7590' same for the web UI)

I have checked that syslog actually send logs event into the file /var/log/udp514.log so I am sure the logs are there. Port 9997 has been allowed on splunk UI (Forwarding and receiving settings).

However  when I do a search : source="/var/log/udp514.log" nothing shows up.

Also splunk throws a message:

'The TCP output processor has paused the data flow. Forwarding to host_dest=andrea-xps-15-7590 inside output group default-autolb-group from host_src=andrea-XPS-15-7590 has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.'

I understand data have been forwarded from host_src but the not indexer for some reason does not ingest them so it get blocked?

Any idea where the problem is?

[John_Littleton]

Hi there,

Most likely 1 of 2 root causes:

1. You do not have indexer configured to received on 9997

https://docs.splunk.com/Documentation/Splunk/9.0.4/Forwarding/Enableareceiver

2. You have a firewall blocking inbound data on 9997 (either firewall on the receiving host, or along the path to it)

Let us know what you find! And an upvote would be appreciated!

[isoutamo]

Hi

you could use e.g. curl to check that the connection from your UF to IDX is open. Just like 

curl -v telnet://<your host name here>:9997
*   Trying [fe80::18e7:b7a4:f93:5eb]:9997...
*   Trying 192.168.0.92:9997...
* Connected to xxxxx (192.168.0.92) port 9997 (#0)

This shows to you that connection has established.

If connection didn't work check @John_Littleton suggestions. One more thing to check if there is SELinux enabled ensure that needed ports etc. are added to it's policy if needed.

r. Ismo 

[richgalloway]

Did you review the receiving system's health like the message suggested?  What did you find out?

Check the firewall(s) between the UF and indexers to make sure connections are allowed.

Also, not the problem, but the inputs.conf stanza should specify an index name.