topic Re: HTTP timed out when setting splunkd-port in Getting Data In

HTTP timed out when setting splunkd-port

mvor — Tue, 19 May 2020 23:32:57 GMT

Hi,

I have a Solaris 11 box, configured with Virtual NIC.
I've installed splunk forwarder, but whenever I try to set port, or list forwarder, I get HTTP timed out:

splunk@serverA:~$ /opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

Couldn't complete HTTP request: Connection timed out

splunk@serverA:~$ /opt/splunkforwarder/bin/splunk list forward-server

Couldn't complete HTTP request: Connection timed out

IP filtering is disabled on this server:

root@serverA:~# svcs ipfilter
STATE STIME FMRI
disabled May_14 svc:/network/ipfilter:default

I'm able to connect to splunk server:

splunk@serverA:~$ telnet splunk 6969

Trying 10.193.10.57...
Connected to splunk.int.rfs.co.nz.
Escape character is '^]'.

I was able to setup splunk forwarder on another serverB, with same OS, and it completed without errors. The only difference between the 2 servers is that serverA has virtual NIC.

Anyone who's encountered this issue? I checked the discussions, and it was pointing to firewall, but firewall is disabled on serverA.

Thanks!

Re: HTTP timed out when setting splunkd-port

codebuilder — Tue, 19 May 2020 23:50:36 GMT

Check your OS configs...telnet is disabled/unavailable by default on almost all Unix flavors, including the antiquated Solaris.
If you have a actual need for telnet on Solaris try the following:

svcadm -v enable -r telnet

Re: HTTP timed out when setting splunkd-port

mvor — Wed, 20 May 2020 00:05:38 GMT

telnet is enabled; and telnet to splunk server to splunkd-port works
splunk@serverA:~$ telnet splunk 6969
Trying 10.193.10.57...
Connected to splunk.int.rfs.co.nz.
Escape character is '^]'.

the issue is i can't set splunkd-port, getting http timed our errors. firewall is disabled.

nothing was modified on the OS config except the addition of virtual nic. tried to configure splunk in another server with same OS but no virtual nic, and it was successful.

Re: HTTP timed out when setting splunkd-port

codebuilder — Wed, 20 May 2020 00:51:36 GMT

Good troubleshooting steps for sure, nice work.
Given your feedback, and looking back over your original post, I think the issue is with your port.

From your post:
splunk@serverA:~$ /opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

That is setting your mgmt port to 6969, which is a secure port, thus not valid for telnet.

Re: HTTP timed out when setting splunkd-port

mvor — Wed, 20 May 2020 01:21:26 GMT

i used that port with the other server, and it worked. also, all our other servers are using that port for splunk.

Re: HTTP timed out when setting splunkd-port

codebuilder — Wed, 20 May 2020 01:27:36 GMT

Hrmm...anything "special" or "different" about this one? Firewall, external NAT, etc. ?

Re: HTTP timed out when setting splunkd-port

mvor — Wed, 20 May 2020 03:04:47 GMT

virtual nic was setup on this server. that's the only difference it has with the other servers.
firewall is disabled.

Re: HTTP timed out when setting splunkd-port

codebuilder — Wed, 20 May 2020 03:13:04 GMT

Well dang...any other differences?
Is this running on physical hardware? If so, is the network switch configured properly?

Have you tried the following?:
svcadm disable network/firewall

Been a long time since I've touched Solaris, so I'm digging here (trying to help)...

Re: HTTP timed out when setting splunkd-port

codebuilder — Wed, 20 May 2020 03:15:19 GMT

I know you mentioned the firewall is disabled already, but the command couldn't hurt again (assuming you are ok with it disabled).

Those aside, I would look outside of Splunk. Cabling, switches, ports, ACL's...

Re: HTTP timed out when setting splunkd-port

mvor — Wed, 20 May 2020 04:06:04 GMT

all machines are virtual.
also tried to disable firewall, but same error.

when i execute this command:
/opt/splunkforwarder/bin/splunk set splunkd-port 6969 --accept-license

does it go out of the server to set the port? i mean, how does the traffic flow with this command?

i appreciate your help.

Re: HTTP timed out when setting splunkd-port

codebuilder — Thu, 21 May 2020 23:42:45 GMT

It doesn't go outside of the server, so to speak, but it has to be available to the other members of the cluster. I think you'll need to open that port on the same subnet that the other members are a part of.

Re: HTTP timed out when setting splunkd-port

mvor — Fri, 29 May 2020 00:53:40 GMT

thanks. the port is already open and is being used by other servers on this subnet. it's only this new server that i can't set splunkd port.

Re: HTTP timed out when setting splunkd-port

PavelP — Fri, 29 May 2020 01:24:28 GMT

Hello @mvor

can you create a network capture with tcpdump/snoop on the Solaris itself? Don't filter by port, just filter out ssh

Re: HTTP timed out when setting splunkd-port

codebuilder — Wed, 30 Sep 2020 05:35:57 GMT

Do you have the port range configured to allow for 6969?

In RHEL / Centos I often have to modify the default range using sysctl:
net.ipv4.ip_local_port_range = 1024 60999

This appears to be the Solaris equivalent (again I'm not a Solaris guy):
https://newbiedba.wordpress.com/2013/01/29/solaris-11-how-to-configure-ephemeral-ports-range-for-oracle-11gr2-installation/