HTTP timed out when setting splunkd-port

[mvor]

Hi,

I have a Solaris 11 box, configured with Virtual NIC.
I've installed splunk forwarder, but whenever I try to set port, or list forwarder, I get HTTP timed out:

splunk@serverA:~$ /opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

Couldn't complete HTTP request: Connection timed out

splunk@serverA:~$ /opt/splunkforwarder/bin/splunk list forward-server

Couldn't complete HTTP request: Connection timed out

IP filtering is disabled on this server:

root@serverA:~# svcs ipfilter
STATE STIME FMRI
disabled May_14 svc:/network/ipfilter:default

I'm able to connect to splunk server:

splunk@serverA:~$ telnet splunk 6969

Trying 10.193.10.57...
Connected to splunk.int.rfs.co.nz.
Escape character is '^]'.

I was able to setup splunk forwarder on another serverB, with same OS, and it completed without errors. The only difference between the 2 servers is that serverA has virtual NIC.

Anyone who's encountered this issue? I checked the discussions, and it was pointing to firewall, but firewall is disabled on serverA.

Thanks!

[PavelP]

Hello @mvor

can you create a network capture with tcpdump/snoop on the Solaris itself? Don't filter by port, just filter out ssh

[codebuilder]

Check your OS configs...telnet is disabled/unavailable by default on almost all Unix flavors, including the antiquated Solaris.
If you have a actual need for telnet on Solaris try the following:

svcadm -v enable -r telnet

[mvor]

telnet is enabled; and telnet to splunk server to splunkd-port works
splunk@serverA:~$ telnet splunk 6969
Trying 10.193.10.57...
Connected to splunk.int.rfs.co.nz.
Escape character is '^]'.

the issue is i can't set splunkd-port, getting http timed our errors. firewall is disabled.

nothing was modified on the OS config except the addition of virtual nic. tried to configure splunk in another server with same OS but no virtual nic, and it was successful.

[codebuilder]

Good troubleshooting steps for sure, nice work.
Given your feedback, and looking back over your original post, I think the issue is with your port.

From your post:
splunk@serverA:~$ /opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

That is setting your mgmt port to 6969, which is a secure port, thus not valid for telnet.

[mvor]

i used that port with the other server, and it worked. also, all our other servers are using that port for splunk.

[codebuilder]

Hrmm...anything "special" or "different" about this one? Firewall, external NAT, etc. ?

[mvor]

virtual nic was setup on this server. that's the only difference it has with the other servers.
firewall is disabled.

[codebuilder]

Well dang...any other differences?
Is this running on physical hardware? If so, is the network switch configured properly?

Have you tried the following?:
svcadm disable network/firewall

Been a long time since I've touched Solaris, so I'm digging here (trying to help)...

[codebuilder]

I know you mentioned the firewall is disabled already, but the command couldn't hurt again (assuming you are ok with it disabled).

Those aside, I would look outside of Splunk. Cabling, switches, ports, ACL's...

[mvor]

all machines are virtual.
also tried to disable firewall, but same error.

when i execute this command:
/opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

does it go out of the server to set the port? i mean, how does the traffic flow with this command?

i appreciate your help.

[codebuilder]

It doesn't go outside of the server, so to speak, but it has to be available to the other members of the cluster. I think you'll need to open that port on the same subnet that the other members are a part of.

[mvor]

thanks. the port is already open and is being used by other servers on this subnet. it's only this new server that i can't set splunkd port.

[codebuilder]

Do you have the port range configured to allow for 6969?

In RHEL / Centos I often have to modify the default range using sysctl:
net.ipv4.ip_local_port_range = 1024 60999

This appears to be the Solaris equivalent (again I'm not a Solaris guy):
https://newbiedba.wordpress.com/2013/01/29/solaris-11-how-to-configure-ephemeral-ports-range-for-ora...