Alerting

How to graph and alert on the state of services from bluepill log data?

ben_siemon_opow
New Member

I use bluepill to track the health of service process (like a java web service for example).

Bluepill outputs lines like:

host service1 Going from starting => up
host service1 Going from up => down
host service1 Going from down => starting
host service1 Going from up => unmonitored

The set of states is small and finite. Some are 'good' states like: 'up' some are 'bad': unmonitored, down.

What mechanisms are there to correlate and track this data?

For example:
1. I would like to display a graph that tracks services by host that transition from up => unmonitored and back often.
2. I would like to alert when a given service transitions too quickly.
3. I would like to see if all services transition to a bad state at the same time.

0 Karma

muebel
SplunkTrust
SplunkTrust

I think the first step would be to extract the fields in the data. i.e.

host <service> Going from <initialstate> => <endstate>

so in props.conf

EXTRACT-bluepill = ^\S+\s(?<service>\S+)\sGoing\sfrom\s(?<initialstate>\S+)\s=>\s(?<endstate>\S+)$

Once you have the fields extracted you can build searches tracking the events that have those fields

search (initialstate=up endstate=down) OR (initialsate=down endstate=starting) | eval end_host = endstate."-".host | timechart count by end_host

Try out various searches utilizes those fields and let me know how it works out.

Also, transaction might be worth checking out : http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Transaction

0 Karma

ben_siemon_opow
New Member

I considered this approach. I was hoping there was feature that was more tailor made for tracking state transitions.

0 Karma

muebel
SplunkTrust
SplunkTrust

It could be the case that the transaction command could help as well : http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Transaction

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...