How to extract events which have zero events?

[vineela]

Hi All,

   I have few events in splunk which will generate all the time, if those events are not generating then we should come to know that there is some issue regarding that. So we have to calculate the events with zero count when checking for data in last 15 mins and display the message in alert stating there are no events in last 15 minutes like that.

Sample Event :

{"log":"[13:18:16.761] [INFO ] [] [c.c.n.t.e.i.T.lloutEventData] [akka://Mmster/user/$b/worrActor/$rOb] - channel=\"AutoNotification\", productVersion=\"2.3.15634ab725\", apiVersion=\"A1\", uuid=\"dee45ca3-2401-13489f240eaf\", eventDateTime=\"2022-09-12T03:18:16.760Z\", severity=\"INFO\", code=\"ServiceCalloutEventData\", component=\"web.client\", category=\"integrational-external\", serviceName=\"Consume Notification\", eventName=\"MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT_REQUEST\", message=\"Schedule Job start, r\", entityType=\"MQST\",returnCode=\"null\"}

I have written query like this:

index=a0_pay MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"|search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification" |stats count by eventName|where count=0|eval message="No Events Triggered for Mandate Notification retreival Callout"|table count message

Not able to fetch results properly... Any other way to find and trigger the results,if there are no evets generated.
Thanks in Advance

[vineela]

i have used same approach but no luck..
index=a0_payservutil_generic_app_audit_prd MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"|search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification" | stats count BY eventName
| append [ | inputlookup mandatecount.csv | eval count=0 | fields eventName count ]
| stats sum(count) AS total BY eventName
|fillnull total
| where total=0

Above is the query is used

[gcusello]

Hi @vineela,

if you have no events, you don't have any result to apply your message, in this case you have to add an empty event, something like this:

index=a0_pay MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*
| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"
| search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification" 
| stats count by eventName
| append [ | makeresults | eval evenName="No events", count=0 | fields eventName count ]
| stats sum(count) AS total By eventName
| where total=0
| eval message="No Events Triggered for Mandate Notification retreival Callout"
| table count message

Ciao.

Giuseppe

[vineela]

Hi Gcusello,

         Thanks a lot for your reply.
But i am not able to view results 
Below is the screenhot:

There is no events in last 40 seconds but i didnt get any message ,If i get any message an alert will trigger.That si the main motto of me.alert should trigger if there are no events in particular time.

[gcusello]

Hi @vineela,

sorry, my mistake a "t" is missing in eventName, please try this:

index=a0_pay MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*
| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"
| search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification" 
| stats count by eventName
| append [ | makeresults | eval eventName="No events", count=0 | fields eventName count ]
| stats sum(count) AS total By eventName
| where total=0
| eval message="No Events Triggered for Mandate Notification retreival Callout"
| table count message

Ciao.

Giuseppe

[vineela]

Hi Sir,

    It is not working sir,,Eventhough there are results in last 15 minutes for that particular event i am able to get message from query.

I should get an alert when there are no results for particular eventName,I the above screenshot if you observe there are more than 100 events for that eventName still i am getting this no events message.It should not happen
I should get an alert only when there are no events for that particular eventName in last 15 minutes

[gcusello]

Hi @vineela,

sorry, probably I diidn't understand you requirement: this solution is to display in a dashboard panel a message instead of "No results".

If you need an alert, you have to use your search without any additional part:

index=a0_pay MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*
| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"
| search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification" 
| stats count by eventName

to trigger when there isn't any result.

Ciao.

Giuseppe

[vineela]

i need to trigger an alert only when the count is "0", I mean no events for that eventName ..I am stuck there

[gcusello]

Hi @vineela,

if you have no results, you can trigger the alert.

when you speak of count=0, I suppose you're meaning the alert condition results=0, so you can run your search.

Ciao.

Giuseppe

[vineela]

Hi Sir,

        Not able to get perfect result. the alert is triggering for every 15 minutes as i gave the timespan of 15 minutes.
Below is the query:
index=a0_payservutil_generic_app_audit_prd MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*
| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"
| search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification"
| stats count by eventName|where count=0|eval message="No events"|fields eventName count message

I tried with |where count<0 also...Same is happening..Can you help me in triggering an alert only when results are zero for such eventname

[gcusello]

Hi @vineela,

probably I am not able to describe the solution:

if you need an alert to trigger when you have no events (count=0) you have to run your search without the condition count=0:

index=a0_payservutil_generic_app_audit_prd MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*
| rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)"
| rex field=log "serviceName=\"*(?<serviceName>[^\"]+)"
| search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification"
| stats count by eventName

in this way you have an alert to trigger when you haven't results (results=0).

You don't need to search the condition count=0 because No results is the condition when count=0 and you cannot display a message, because with the message you have one solution.

If instead you want to dispay the result in a panel and you want to manage the condition "No results" displaying your message, you can use my first solution to display a message.

You cannot have an intermediate solution, what do you need?

• an alert: the above solution,
• a dashboard panel: my first solution.

Ciao.

Giuseppe

[bowesmana]

If you have events for event name X then there are no events to get the name X from the event, so you will never have 

X = 0

because X does not exist.

There are a number of ways of solving this 'missing' problem, search the community, e.g.

https://community.splunk.com/t5/Splunk-Search/Identify-missing-events-based-Lookup-list-of-values/m-...

https://community.splunk.com/t5/Alerting/How-to-trigger-alert-if-host-is-missing-reporting-based-on/...

You can either do this with a lookup where the lookup contains a list of all the 'required' event names or you can use something like the TrackMe app

https://splunkbase.splunk.com/app/4621/