Getting Data In

Splunk Forwarder Unable to communicate with Server

user789
New Member

I am running RHEL 7 server, and noticed that my splunk forwarder client is not reporting in. I am running iptables. Here are the rules that I've added:
-A INPUT -p tcp -m tcp --dport 8089 -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 8089 -j ACCEPT

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The source of data from a forwarder will not be port 8089. It will be an arbitrary port starting near 65000.
Data from a forwarder is sent to port 9997 (or whatever port your indexer listens on).
Forwarders will use the deployment server's port 8089 when phoning home.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

chandralinginen
Observer

I believe you gave it in the reverse. Try OUTPUT dport 8089 and restart splunk

0 Karma

chandralinginen
Observer

@user789 So, let me give you some more information to what I think. Each UF is also a splunk installation. It will have a management port 8089 running. That is only used when you try connecting to the UF remotely and perform splunk commands. You can disable the management port on the UF, no one cares.

Secondly, are you referring to the Forwarder Management on the Deployment Server? If this is the place you are not seeing your UF, then, just enable iptables rule for OUTPUT from your UF ( ANY to ANY ) on port 8089/TCP. This will make the UF to be able to pass the communication to the DS. And the DS listens on port 8089 which is the DS's management port.
And thirdly, Have you given the DS info? You can do this via an app in etc\apps\customname\local\deployment.conf

If you have done all the above and splunk service is running you should see the UF as long as your DS is also accepting 8089 Communications. And if you see it on the DS and data is not coming to your indexer, let us know.

0 Karma

user789
New Member

The machine is not showing up on the Deployment server.
I switched the iptables rules around, and it still does not report in.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The source of data from a forwarder will not be port 8089. It will be an arbitrary port starting near 65000.
Data from a forwarder is sent to port 9997 (or whatever port your indexer listens on).
Forwarders will use the deployment server's port 8089 when phoning home.

---
If this reply helps you, Karma would be appreciated.

user789
New Member

When I do a ps -aux, I see splunkd only using port 8089. I believe I used the defaults during the install.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Which instance of Splunk did you look at? The forwarder should be allowed to send from any port to port 9997 on your indexer instance(s) and to port 8089 on your deployment server (if you have one).

---
If this reply helps you, Karma would be appreciated.
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @user789,
what do you mean with "Unable to communicate with Server"?
If you're speaking of management communication, it works on port 8089.
Data send instead, by default, uses the 9997 port.
Did you verified that from the UF you can reach the Indexer on port 9997 (or the one you're using)?
You can check this with telnet from UF:

telnet ip_indexer 9997

Then did you enabled log receinving on port 9997 (or another one)?
to do this, by GUI: [Settings -- Forwarding and Receiving -- Receive data -- Configure receiving -- New receiving port].

Ciao.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @user789,
as I said, did you enabled receiving on port 9997 on the Indexer?
then you can check if the route between Universal Forwarder and Indexer is open on port 9997 using telnet.

Ciao.
Giuseppe

0 Karma

user789
New Member

Yes.
It said connection refused.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @user789,
at first check receiving on port 9997 on Indexer,
then check the iptables on Indexer (input) and Universal Forwarder (output) on port 9997.
then eventual other firewalls in the middle.

At least, did you configured outputs.conf (or using the command ./splunk add forward-server <host name or ip address>:<listening port> )?

Ciao.
Giuseppe

0 Karma

user789
New Member

I know that the Indexer is receiving logs from other machines.
When I try to add this forward-server, it tells me the login failed.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @user789,
this is probably a different problem.

At first check that the route is open using telnet from the UF to the Indexer on the port you enabled on Indexer (default 9997).
then you have to configure your UF to send logs to your indexer.
You can do this in two different ways:

  • modifying outputs.conf (copy it from another Uf that's running);
  • using the command ./splunk add forward-server : , but if you have login failed, probably you don't remember the admin password on UF.

in this second case, uninstall and reinstall Splunk UF.

Ciao.
Giuseppe

0 Karma

user789
New Member

After adding this rule, I was able to telnet to my server on port 9997: iptables -I OUTPUT -p tcp --dport 9997 -j ACCEPT

0 Karma

user789
New Member

I think adding this iptables rule fixed it!

0 Karma

chandralinginen
Observer

add 8089 in the same way and it should work as well. if it does. confirm the answer. thanks!

0 Karma

user789
New Member

I can disable 8089, right?
I still can't add the forwarder using the command. It would be the login creds that I use to login to indexer right?

0 Karma

chandralinginen
Observer

you can disable 8089 on the UF. And if you are executing he command on the UF, then the creds are what you used for installation or the default ones if you havent changed (admin/changeme)

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @user789,
ok, now you have to configure you UF to send data to the indexer.
You can copy in it the outputs.conf of another UF and then restart Splunk.

So you can see the internal logs of the UF in your Splunk, running

index=_internal host=UF_hostname

Ciao.
Giuseppe

0 Karma

chandralinginen
Observer

If it says connection refused, then port 9997 is not open on your indexer, or there is not splunk service listening on port 9997.

0 Karma

chandralinginen
Observer

I believe its just the uptable. You should add the output connection for dst port 8089 from your server and try. you did it in reverse previously.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...