Getting Data In

Why are my Universal Forwarder logs not reaching my Indexer?

johannterc
New Member

I'm troubleshooting why my Splunk Universal Forwarder (UF) logs in Active Directory Forest B are not reaching my Splunk indexer which is located in AD Forest A. TCP 9997 has been opened up in the firewall between the two forests already. Are there logs in the UF install folder located on the Splunk UF that would shed some light? Or should I be looking somewhere else?

0 Karma

jrballesteros05
Communicator

There can be many reasons:

  • Firewall: You said you opened the ports and I believe you, so I would discard this one.
  • Internal Firewall: You did not mention if you are running splunk over Windows or Linux but I am more experienced on Linux and I normally like configuring my own firewall in the Linux Server, especially in Centos. So, you should check if your Indexer is not running a firewall.

If your Indexer is running on Linux I recommend to check port with telnet from the Universal forwarder (You are talking about Active directory so I guest is a Windows Server) :

telnet yourindexerip 9997

You can also check connections from any Unix OS with nmap:

nmap -Pn yourindexerip -p 9997
  • Routing: Sometimes you cannot reach the indexer because of routing tables, in my opinion we only have to configure the default gateway but sometimes for many reasons people configure static routes in servers, so I think you should run a traceroute or tracert between the Windows Server and the Indexer.

    tracert yourindexerip

If any of the reasons above cannot help you, you should be more specific and we try to help you. I'm guessing your problem is communication but can be something else.

Best regards.

0 Karma

johannterc
New Member

Hello jrballesteros05. Do you know if the port has to be open and listening from both ends? We have the firewall configured to only have the port listening from the indexer and NOT listening on the UF....

0 Karma

johannterc
New Member

(Forgot to add that telnet from the UF to the indexer works successfully)

0 Karma

jrballesteros05
Communicator

The indexer must be listening in the 9997/TCP port, and the UF usually uses a random port to conect to the indexer.

You can check this in the indexer with this command (I'm assuming you are running your indexer over Linux):

 ss -putan | grep 9997 

You should get something like this:

 root@myindexer~# ss -putan | grep 9997
 tcp    LISTEN     0      128                    *:9997                  *:*      users:(("splunkd",28307,42))
 tcp    **ESTAB**      0      0           yourindexerip:9997       youruniversalforwarderip:**33809**  users:(("splunkd",28307,185))

In my case the port "33809" is any port that UF takes to connect with the indexer. You also may have your connection right and you are not seeing the UF in the deployment server because the connection is in the 8089TCP, maybe you are receiving logs but you are not be able to control the UF remotely.

0 Karma

adonio
Ultra Champion

Hi johannterc,
can you elaborate a little, when searching index=_internal host=YourADForwarder do you see data?
do you have outputs.conf setup on your forwarder?

0 Karma

johannterc
New Member

Hello Adonio. Not sure if I have outputs.conf setup on the forwarder. Are you referring to the heavy forwarder?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...