Monitoring Splunk

How to monitor Application in OpenShift Cluster with Splunk?

noyog
Engager

Hi,

My application is running on OpenShift pods.
The application accepts API calls on port 9443.
In front of the pods, we have ELB which is the public entry point into the application.
The ELB accepts traffic on port 443 and pass them through to the pods over 9443 (it doesn't terminate TLS).

My Splunk server is external to the OpenShift cluster.
We created a Splunk add-on that performs API calls to the application (it works when the application is installed on VM).

When we try to monitor the application behind OpenShift, Splunk gets a 503 (Application is not available) response from OpenShift.
If we do an API call via CURL command from the Splunk server, the call is successful so we know the routing from the ELB to the OpenShift pods works.
Interestingly, when we change the URL in the CURL command to be one of the ELB IPs, we reproduce the issue we see in Splunk.

Is there any special configuration we need to perform in order for the monitoring to work against OpenShift?

0 Karma
1 Solution

noyog
Engager

We managed to resolve the issue.
Our add-on was built using Splunk's add-on builder and that comes with httplib2 v0.9.2.
That version doesn't support SNI and as a result, the API calls from the add-on to the OpenShift cluster were answered with 503 ('The host doesn't exist' error).
It seems like OpenShift requires SNI so the pods know how to direct the calls.

We manually updated the httplib2 version in the add-on and it resolved the issue.
Any httplib2 version above 0.9.2 supports SNI.

CURL supports SNI which is why those calls worked.

View solution in original post

0 Karma

noyog
Engager

We managed to resolve the issue.
Our add-on was built using Splunk's add-on builder and that comes with httplib2 v0.9.2.
That version doesn't support SNI and as a result, the API calls from the add-on to the OpenShift cluster were answered with 503 ('The host doesn't exist' error).
It seems like OpenShift requires SNI so the pods know how to direct the calls.

We manually updated the httplib2 version in the add-on and it resolved the issue.
Any httplib2 version above 0.9.2 supports SNI.

CURL supports SNI which is why those calls worked.

0 Karma
Get Updates on the Splunk Community!

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...

Splunk AppDynamics Agents Webinar Series

Mark your calendars! On June 24th at 12PM PST, we’re going live with the second session of our Splunk ...

SplunkTrust Application Period is Officially OPEN!

It's that time, folks! The application/nomination period for the 2025 SplunkTrust is officially open! If you ...