Splunk Forwarder Unable to communicate with Server

[user789]

I am running RHEL 7 server, and noticed that my splunk forwarder client is not reporting in. I am running iptables. Here are the rules that I've added:
-A INPUT -p tcp -m tcp --dport 8089 -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 8089 -j ACCEPT

[chandralinginen]

I believe you gave it in the reverse. Try OUTPUT dport 8089 and restart splunk

[chandralinginen]

@user789 So, let me give you some more information to what I think. Each UF is also a splunk installation. It will have a management port 8089 running. That is only used when you try connecting to the UF remotely and perform splunk commands. You can disable the management port on the UF, no one cares.

Secondly, are you referring to the Forwarder Management on the Deployment Server? If this is the place you are not seeing your UF, then, just enable iptables rule for OUTPUT from your UF ( ANY to ANY ) on port 8089/TCP. This will make the UF to be able to pass the communication to the DS. And the DS listens on port 8089 which is the DS's management port.
And thirdly, Have you given the DS info? You can do this via an app in etc\apps\customname\local\deployment.conf

If you have done all the above and splunk service is running you should see the UF as long as your DS is also accepting 8089 Communications. And if you see it on the DS and data is not coming to your indexer, let us know.

[user789]

The machine is not showing up on the Deployment server.
I switched the iptables rules around, and it still does not report in.

[richgalloway · Accepted solution]

The source of data from a forwarder will not be port 8089. It will be an arbitrary port starting near 65000.
Data from a forwarder is sent to port 9997 (or whatever port your indexer listens on).
Forwarders will use the deployment server's port 8089 when phoning home.

[user789]

When I do a ps -aux, I see splunkd only using port 8089. I believe I used the defaults during the install.

[richgalloway]

Which instance of Splunk did you look at? The forwarder should be allowed to send from any port to port 9997 on your indexer instance(s) and to port 8089 on your deployment server (if you have one).

[gcusello]

Hi @user789,
what do you mean with "Unable to communicate with Server"?
If you're speaking of management communication, it works on port 8089.
Data send instead, by default, uses the 9997 port.
Did you verified that from the UF you can reach the Indexer on port 9997 (or the one you're using)?
You can check this with telnet from UF:

telnet ip_indexer 9997

Then did you enabled log receinving on port 9997 (or another one)?
to do this, by GUI: [Settings -- Forwarding and Receiving -- Receive data -- Configure receiving -- New receiving port].

Ciao.
Giuseppe

[gcusello]

Hi @user789,
as I said, did you enabled receiving on port 9997 on the Indexer?
then you can check if the route between Universal Forwarder and Indexer is open on port 9997 using telnet.

Ciao.
Giuseppe

[user789]

Yes.
It said connection refused.

[gcusello]

Hi @user789,
at first check receiving on port 9997 on Indexer,
then check the iptables on Indexer (input) and Universal Forwarder (output) on port 9997.
then eventual other firewalls in the middle.

At least, did you configured outputs.conf (or using the command ./splunk add forward-server <host name or ip address>:<listening port> )?

Ciao.
Giuseppe

[user789]

I know that the Indexer is receiving logs from other machines.
When I try to add this forward-server, it tells me the login failed.

[gcusello]

Hi @user789,
this is probably a different problem.

At first check that the route is open using telnet from the UF to the Indexer on the port you enabled on Indexer (default 9997).
then you have to configure your UF to send logs to your indexer.
You can do this in two different ways:

• modifying outputs.conf (copy it from another Uf that's running);
• using the command ./splunk add forward-server : , but if you have login failed, probably you don't remember the admin password on UF.

in this second case, uninstall and reinstall Splunk UF.

Ciao.
Giuseppe

[user789]

After adding this rule, I was able to telnet to my server on port 9997: iptables -I OUTPUT -p tcp --dport 9997 -j ACCEPT

[user789]

I think adding this iptables rule fixed it!

[chandralinginen]

add 8089 in the same way and it should work as well. if it does. confirm the answer. thanks!

[user789]

I can disable 8089, right?
I still can't add the forwarder using the command. It would be the login creds that I use to login to indexer right?

[chandralinginen]

you can disable 8089 on the UF. And if you are executing he command on the UF, then the creds are what you used for installation or the default ones if you havent changed (admin/changeme)

[gcusello]

Hi @user789,
ok, now you have to configure you UF to send data to the indexer.
You can copy in it the outputs.conf of another UF and then restart Splunk.

So you can see the internal logs of the UF in your Splunk, running

index=_internal host=UF_hostname

Ciao.
Giuseppe

[chandralinginen]

If it says connection refused, then port 9997 is not open on your indexer, or there is not splunk service listening on port 9997.

[chandralinginen]

I believe its just the uptable. You should add the output connection for dst port 8089 from your server and try. you did it in reverse previously.