Splunk Monitoring

[gdavid]

I want to have a check for Splunk by monitoring it with our monitoring platform.
We recently ran into a windows bug where we run out of server connections. Server is pingable, splunkd/splunk web are both running fine.

can someone provide a more creative way to monitor?

during our bug i noticed that splunk web was unable to populate the login page with proper footer info:

© 2005-2013 Splunk Inc. Splunk 5.0.0 build 0000000.

it came up as version unknown. this was one additional way i was considering monitoring it.

[sowings]

When you see that version unknown, the splunkweb (UI) process was unable to contact splunkd (Splunk service).

[sowings]

Are you using the Splunk deployment server while having it be a client of itself? I saw the issue linked below when attempting to do the same:

http://answers.splunk.com/answers/41449/winsock-error-10055

[lukejadamec]

When you say you ran out of server connections what do you mean?
You say splunk web was running fine, but the page did not fully populate... were you able to log in? Were you on the local system or accessing it from the network?
You said that splunkd was running, but was it working? Were the forwarders able to send data, or were the forwarder logs showing connection errors?