Getting Data In

HTTP timed out when setting splunkd-port

mvor
Explorer

Hi,

I have a Solaris 11 box, configured with Virtual NIC.
I've installed splunk forwarder, but whenever I try to set port, or list forwarder, I get HTTP timed out:

splunk@serverA:~$ /opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

Couldn't complete HTTP request: Connection timed out

splunk@serverA:~$ /opt/splunkforwarder/bin/splunk list forward-server

Couldn't complete HTTP request: Connection timed out

IP filtering is disabled on this server:

root@serverA:~# svcs ipfilter
STATE STIME FMRI
disabled May_14 svc:/network/ipfilter:default

I'm able to connect to splunk server:

splunk@serverA:~$ telnet splunk 6969

Trying 10.193.10.57...
Connected to splunk.int.rfs.co.nz.
Escape character is '^]'.

I was able to setup splunk forwarder on another serverB, with same OS, and it completed without errors. The only difference between the 2 servers is that serverA has virtual NIC.

Anyone who's encountered this issue? I checked the discussions, and it was pointing to firewall, but firewall is disabled on serverA.

Thanks!

Tags (1)
0 Karma

PavelP
Motivator

Hello @mvor

can you create a network capture with tcpdump/snoop on the Solaris itself? Don't filter by port, just filter out ssh

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Check your OS configs...telnet is disabled/unavailable by default on almost all Unix flavors, including the antiquated Solaris.
If you have a actual need for telnet on Solaris try the following:

svcadm -v enable -r telnet
----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

mvor
Explorer

telnet is enabled; and telnet to splunk server to splunkd-port works
splunk@serverA:~$ telnet splunk 6969
Trying 10.193.10.57...
Connected to splunk.int.rfs.co.nz.
Escape character is '^]'.

the issue is i can't set splunkd-port, getting http timed our errors. firewall is disabled.

nothing was modified on the OS config except the addition of virtual nic. tried to configure splunk in another server with same OS but no virtual nic, and it was successful.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Good troubleshooting steps for sure, nice work.
Given your feedback, and looking back over your original post, I think the issue is with your port.

From your post:
splunk@serverA:~$ /opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

That is setting your mgmt port to 6969, which is a secure port, thus not valid for telnet.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

mvor
Explorer

i used that port with the other server, and it worked. also, all our other servers are using that port for splunk.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Hrmm...anything "special" or "different" about this one? Firewall, external NAT, etc. ?

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

mvor
Explorer

virtual nic was setup on this server. that's the only difference it has with the other servers.
firewall is disabled.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Well dang...any other differences?
Is this running on physical hardware? If so, is the network switch configured properly?

Have you tried the following?:
svcadm disable network/firewall

Been a long time since I've touched Solaris, so I'm digging here (trying to help)...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

I know you mentioned the firewall is disabled already, but the command couldn't hurt again (assuming you are ok with it disabled).

Those aside, I would look outside of Splunk. Cabling, switches, ports, ACL's...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

mvor
Explorer

all machines are virtual.
also tried to disable firewall, but same error.

when i execute this command:
/opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

does it go out of the server to set the port? i mean, how does the traffic flow with this command?

i appreciate your help.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

It doesn't go outside of the server, so to speak, but it has to be available to the other members of the cluster. I think you'll need to open that port on the same subnet that the other members are a part of.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

mvor
Explorer

thanks. the port is already open and is being used by other servers on this subnet. it's only this new server that i can't set splunkd port.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Do you have the port range configured to allow for 6969?

In RHEL / Centos I often have to modify the default range using sysctl:
net.ipv4.ip_local_port_range = 1024 60999

This appears to be the Solaris equivalent (again I'm not a Solaris guy):
https://newbiedba.wordpress.com/2013/01/29/solaris-11-how-to-configure-ephemeral-ports-range-for-ora...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...