Splunk Search

Help with Heartbeat check

metylkinandrey
Communicator

Good afternoon!
I have six Heartbeat messages coming from the system. All messages from the chain are connected by one: "srcMsgId". Messages have a certain interval, if the interval between messages is higher, that is, one message in a chain (out of six messages) is late, say six seconds (the normal interval is five seconds), then an alarm is triggered. Can you tell me how to do it?
I tried something like this but it's not exactly what you need:

index="bl_logging" sourcetype="testsystem-2" srcMsgId="rwfsdfsfqwe121432gsgsfgd80"

| transaction maxpause=5m srcMsgId Correlation_srcMsgId messageId

| table _time srcMsgId Correlation_srcMsgId messageId duration eventcount

| sort srcMsgId _time

| streamstats current=f window=1 values(_time) as prevTime by subject

| eval timeDiff=_time-prevTime

| delta _time as timediff

| where (timediff)>6

This will only show me the lagging message.

Messages arrive one after another, so we can see their interval and, in theory, take this opportunity to create an alarm when the interval is increased.

Please tell me how can I do this.
Alas I don't selenium in splunk.

 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

HI @metylkinandrey,

please try something like this:

index="bl_logging" sourcetype="testsystem-2" srcMsgId="rwfsdfsfqwe121432gsgsfgd80"
| transaction maxevents=2 srcMsgId Correlation_srcMsgId messageId
| where duration>5
| table _time srcMsgId Correlation_srcMsgId messageId duration eventcount

Ciao.

Giuseppe

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Event Series: Splunk Observability Metrics Cost Optimization

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...

Deep insights, no barriers: Splunk Observability Cloud Free Edition

As software delivery cycles continue to accelerate, observability shouldn’t be a luxury — it should be a ...