topic Re: splunk forwarder not detecting in Splunk Search

splunk forwarder not detecting

okumar1 — Thu, 13 Mar 2025 09:07:05 GMT

Hello everyone,

I have set up my Splunk server[with receiving port 9997 is enabled] and Splunk forwarder to monitor my UF logs. Please suggest what i am missing here.

but i am getting below when i do - ./splunk list forward-server

o/p:

Active forwards:
None
Configured but inactive forwards:
52.66.100.58:9997

i have done below steps:

my UF: ./splunk add forward-server 52.66.100.58:9997

outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = 52.66.100.58:9997

[tcpout-server://52.66.100.58:9997]

Thanks in advance.

Re: splunk forwarder not detecting

gcusello — Thu, 13 Mar 2025 09:54:14 GMT

Hi @okumar1 ,

at first, did you checked if the connection (firewall route) is open between UF and IDX on the 9997 port?

you can check this on UF using telnet.

Then, is the 9997 port open on the IDX local firewall?

Ciao.

Giuseppe

Re: splunk forwarder not detecting

livehybrid — Thu, 13 Mar 2025 10:04:56 GMT

Hi @okumar1

I see you have put a screenshot of the rule that allows inbound traffic to your server, but is your UF server also configured with outbound connectivity on port 9997?

If you are using linux with netcat installed then this might work well to test:

nc -vz -w1 yourServerIP 9997 you can also check with portquiz.net nc -vz -w1 portquiz.net 9997

The portquiz.net test relies on you being able to reach the internet from your server.

Let us know how you get on and we can investigate further.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

Re: splunk forwarder not detecting

okumar1 — Fri, 14 Mar 2025 05:54:08 GMT

Hi @gcusello,

as per your comment, yes my UF has outbound rules set on port 9997, still not working. Please suggest

Re: splunk forwarder not detecting

livehybrid — Fri, 14 Mar 2025 06:49:24 GMT

Hi @okumar1

Were you able to check the netcat/nc commands?

Are there any other logs around mentioning tcpOutput in your UF?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

Re: splunk forwarder not detecting

gcusello — Fri, 14 Mar 2025 06:55:54 GMT

Hi @okumar1 ,

what about telnet test?

Ciao.

Giuseppe

Re: splunk forwarder not detecting

okumar1 — Fri, 14 Mar 2025 07:22:15 GMT

here is the output

[root@ip-172-31-13-139 log]# nc -vz -w1 13.233.165.44 9997
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

no my outputs.conf is below:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 13.233.165.44:9997

and when i debug splunkd.log

The TCP output processor has paused the data flow. Forwarding to host_dest=13.233.165.44 inside output group default-autolb-group from host_src=ip-172-31-13-139.ap-south-1.compute.internal has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data

please suggest

Re: splunk forwarder not detecting

okumar1 — Fri, 14 Mar 2025 07:25:15 GMT

hi @gcusello ,

here is the telnet test

telnet: connect to address 13.233.165.44: Connection refused

and splund.log

please suggest.

Re: splunk forwarder not detecting

gcusello — Fri, 14 Mar 2025 07:49:30 GMT

Hi @okumar1 ,

this means that there's something in the middle between UD and IDX that block the connection.

probably an intermediate firewall or a local firewall on the IDX.

Ciao.

Giuseppe