How to monitor Application in OpenShift Cluster with Splunk?

[noyog]

Hi,

My application is running on OpenShift pods.
The application accepts API calls on port 9443.
In front of the pods, we have ELB which is the public entry point into the application.
The ELB accepts traffic on port 443 and pass them through to the pods over 9443 (it doesn't terminate TLS).

My Splunk server is external to the OpenShift cluster.
We created a Splunk add-on that performs API calls to the application (it works when the application is installed on VM).

When we try to monitor the application behind OpenShift, Splunk gets a 503 (Application is not available) response from OpenShift.
If we do an API call via CURL command from the Splunk server, the call is successful so we know the routing from the ELB to the OpenShift pods works.
Interestingly, when we change the URL in the CURL command to be one of the ELB IPs, we reproduce the issue we see in Splunk.

Is there any special configuration we need to perform in order for the monitoring to work against OpenShift?

[noyog · Accepted solution]

We managed to resolve the issue.
Our add-on was built using Splunk's add-on builder and that comes with httplib2 v0.9.2.
That version doesn't support SNI and as a result, the API calls from the add-on to the OpenShift cluster were answered with 503 ('The host doesn't exist' error).
It seems like OpenShift requires SNI so the pods know how to direct the calls.

We manually updated the httplib2 version in the add-on and it resolved the issue.
Any httplib2 version above 0.9.2 supports SNI.

CURL supports SNI which is why those calls worked.