Is there a Health Check for external HTTP Event Forwarder?

[mninansplunk]

Hello,

I have been asked to monitor our HTTP Event Forwarder.  Is there a Health Check in Splunk that would tell me the Forwarder status?  Or is there another way I could view if the Event Forwarder is down without going into Splunk Enterprise?  Perhaps a URL that would simply give me an HTTP Status code or something.

Thanks,

Tom

[PickleRick]

Depends on what kind of check you want to have.

You can, for example:

• Check for an open port
• Check for valid TLS certificate and possibly TLS authentication (if you're using it)
• Check for error response if you supply a wrongly composed HEC request (to a non-existant endpoint or posting badly-formed data)
• Post an event and check if it's getting indexed properly (this one of course needs an access to search-head). Alternatively you could configure your splunk to just route the event somewhere "to the outside" instead of indexing it so you could monitor the receiving party for the event.
• Post an event and do a scheduled search which will report if the event has been indexed (this one doesn't require direct access to the splunk console/API for the monitoring purposes but you have to be able to receive output from the report via means of email or some custom alertaction.

So you see - there are many aspects to monitoring. First three checks give you some form of "technical" monitoring. The fourth and fifth ones monitor whole process (or at least part of it if you chose to push the event off the splunk environment in the end).

[mninansplunk]

Thanks PickleRick,  (Great Name, Great Show).  I am wanting to utilize the Splunk API to verify that the Forwarder Health is Ok,  That the Forwarder is up.  But, I'm having a hard time finding an API that contains the Forwarder.  Is there an API for the Forwarder that would give green/yellow/red statuses?

Thanks,

Tom

[PickleRick · Accepted solution]

I never used it myself but I suppose you can query server/health* endpoints on your HF. This might give you what you need. For a health status of a specific hec input you may try services/collector/health endpoint.

Full reference of REST endpoints: https://docs.splunk.com/Documentation/Splunk/8.2.2/RESTREF/RESTlist