splunk forwarder not detecting

[okumar1]

Hello everyone,

I have set up my Splunk server[with receiving port 9997 is enabled] and Splunk forwarder to monitor my UF logs.  Please suggest what i am missing here.

but i am getting below when i do - ./splunk list forward-server

o/p:

Active forwards:
None
Configured but inactive forwards:
52.66.100.58:9997

i have done below steps:

my UF:  ./splunk add forward-server 52.66.100.58:9997

outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = 52.66.100.58:9997

[tcpout-server://52.66.100.58:9997]

Thanks in advance.

[livehybrid]

Hi @okumar1 

I see you have put a screenshot of the rule that allows inbound traffic to your server, but is your UF server also configured with outbound connectivity on port 9997?

If you are using linux with netcat installed then this might work well to test:

nc -vz -w1 yourServerIP 9997

you can also check with portquiz.net
nc -vz -w1 portquiz.net 9997

The portquiz.net test relies on you being able to reach the internet from your server.

Let us know how you get on and we can investigate further.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[okumar1]

Hi @gcusello,

as per your comment, yes my UF has outbound rules set on port 9997, still not working. Please suggest

[gcusello]

Hi @okumar1 ,

what about telnet test?

Ciao.

Giuseppe

[okumar1]

hi @gcusello ,

here is the telnet test

telnet: connect to address 13.233.165.44: Connection refused

and splund.log

please suggest.

[gcusello]

Hi @okumar1 ,

this means that there's something in the middle between UD and IDX that block the connection.

probably an intermediate firewall or a local firewall on the IDX.

Ciao.

Giuseppe

[livehybrid]

Hi @okumar1 

Were you able to check the netcat/nc commands? 

Are there any other logs around mentioning tcpOutput in your UF?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[okumar1]

here is the output

[root@ip-172-31-13-139 log]# nc -vz -w1 13.233.165.44 9997
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

no  my outputs.conf is below:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 13.233.165.44:9997

and when i debug splunkd.log

The TCP output processor has paused the data flow. Forwarding to host_dest=13.233.165.44 inside output group default-autolb-group from host_src=ip-172-31-13-139.ap-south-1.compute.internal has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data

please suggest

[gcusello]

Hi @okumar1 ,

at first, did you checked if the connection (firewall route) is open between UF and IDX on the 9997 port?

you can check this on UF using telnet.

Then, is the 9997 port open on the IDX local firewall?

Ciao.

Giuseppe