Why are my Universal Forwarder logs not reaching my Indexer?

[johannterc]

I'm troubleshooting why my Splunk Universal Forwarder (UF) logs in Active Directory Forest B are not reaching my Splunk indexer which is located in AD Forest A. TCP 9997 has been opened up in the firewall between the two forests already. Are there logs in the UF install folder located on the Splunk UF that would shed some light? Or should I be looking somewhere else?

[jrballesteros05]

There can be many reasons:

• Firewall: You said you opened the ports and I believe you, so I would discard this one.
• Internal Firewall: You did not mention if you are running splunk over Windows or Linux but I am more experienced on Linux and I normally like configuring my own firewall in the Linux Server, especially in Centos. So, you should check if your Indexer is not running a firewall.

If your Indexer is running on Linux I recommend to check port with telnet from the Universal forwarder (You are talking about Active directory so I guest is a Windows Server) :

telnet yourindexerip 9997

You can also check connections from any Unix OS with nmap:

nmap -Pn yourindexerip -p 9997

• Routing: Sometimes you cannot reach the indexer because of routing tables, in my opinion we only have to configure the default gateway but sometimes for many reasons people configure static routes in servers, so I think you should run a traceroute or tracert between the Windows Server and the Indexer.

  tracert yourindexerip

If any of the reasons above cannot help you, you should be more specific and we try to help you. I'm guessing your problem is communication but can be something else.

Best regards.

[johannterc]

Hello jrballesteros05. Do you know if the port has to be open and listening from both ends? We have the firewall configured to only have the port listening from the indexer and NOT listening on the UF....

[johannterc]

(Forgot to add that telnet from the UF to the indexer works successfully)

[jrballesteros05]

The indexer must be listening in the 9997/TCP port, and the UF usually uses a random port to conect to the indexer.

You can check this in the indexer with this command (I'm assuming you are running your indexer over Linux):

 ss -putan | grep 9997 

You should get something like this:

 root@myindexer~# ss -putan | grep 9997
 tcp    LISTEN     0      128                    *:9997                  *:*      users:(("splunkd",28307,42))
 tcp    **ESTAB**      0      0           yourindexerip:9997       youruniversalforwarderip:**33809**  users:(("splunkd",28307,185))

In my case the port "33809" is any port that UF takes to connect with the indexer. You also may have your connection right and you are not seeing the UF in the deployment server because the connection is in the 8089TCP, maybe you are receiving logs but you are not be able to control the UF remotely.

[adonio]

Hi johannterc,
can you elaborate a little, when searching index=_internal host=YourADForwarder do you see data?
do you have outputs.conf setup on your forwarder?

[johannterc]

Hello Adonio. Not sure if I have outputs.conf setup on the forwarder. Are you referring to the heavy forwarder?