Resolved: SSL error connecting on port 8089 from server

[ahmadshabbir]

I was getting SSL error due to a self signed certificate on port 8089. This certificate has been replaced with a DigiCert signed certificate. With updated certificate I am able to connect to Splunk API on 8089 from my local desktop. However, when I am still getting SSL error when connecting from our application server. I validated that ports 8089 and 443 is open from app server. I get to Splunk on port 443 from app server but when trying to connect on port 8089, getting SSL error. 

Please help me what could be causing this and how to resolve this error.

[codebuilder]

How are you testing the connection?

And is your app server behind a load balancer?

If so, you'll likely need a second IP / DNS name and route calls from that IP on port 443 to 8089 on your search heads, and have your application use that for API calls.

[ahmadshabbir]

I am not sure how are you suggesting to route the call on 443 to 8089. Current Development env app server is standalone. It however is behind our firewall and IP is NATed. We have validated that firewall is not an blocker. 

Please elaborate the option.

[codebuilder]

If you are using a load balancer you can only route 443 to a single destination.

I'm guessing that you want to expose the Splunk UI to end users and use https.

The Splunk UI by default runs on port 8000 (which you can change), but unless you want the URL to your Splunk applications to include a specific port, you have to route https (443) to 8000 on the search heads.

Otherwise, your URL would have to be something like https://www.mysplunkserver.com:8000.

You want your API calls to be secure as well, but you can't route port 443 from the same IP to two different destinations. The second destination will never be reached because it would be satisfied by the first rule.

The solution / best practice, is to have one IP to expose your UI to end users on https (443 to 8000) and a second IP for API calls and behind the scenes Splunk traffic (443 to 8089).

Though not required necessarily, I generally assign a DNS name to the IP for the API. That way if the IP needs to change your only have to update DNS, and not your code.

[ahmadshabbir]

Thank you for the details. In my case, I am not managing Splunk, I am consuming our Splunk services from my application.  The Splunk team provided the URL https://mycorp.splunkcloud.com:8089/services/search/jobs/

It is working perfectly fine from my desktop. However, when trying to access this from an application (running within same firewall), I am getting an SSL error. We have added Splunk certificate in app trusted store but still we are getting same error.

[ahmadshabbir]

It turn out to be a firewall issue. Even though port 8089 was allowed, the SSL traffic was being blocked,