Alerting

Can you help me create an alert involving the transaction command?

raj_mpl
Path Finder

Hi All ,

I wanted to know exactly what my below query will generate . Also please explain about keeporphans and keepevicted

index=myindex   | rex field=_raw "sometext\s(?[^\)]+)\s\((?[^\)]+)" | rex field=_raw "\[sometext:\s(?[^\]]+)" | transaction  myfield , id , host  startswith="started " endswith="completed" keepevicted=true | where closed_txn=0 | search NOT stopped  | table host, _time, myfield, id

What I understood is, by keepevicted=true , the events that are not satisfying the transaction command properties will be visible.

I found duration field value as 0 for the above query, but my requirement is, I want to create an alert if the myfield is running more than 30 min.

Thank you !

Tags (2)
0 Karma

renjith_nair
SplunkTrust
SplunkTrust

@raj_mpl ,

Continuing from the other question, how to populate another filed if search time extracted filed, if you have uniq fileds for each job,transaction, avoid transaction command and try something like below

index=myindex ("started plan instance") OR ("successfully completed Plan")
   |"all your rex to extract Job_Name,id  and other fields"
   |stats count,earliest(_time) as _time by id,Job_Name
   |where count <2|eval duration=round((now()-_time)/3600,2)|where duration>2
Happy Splunking!
0 Karma

raj_mpl
Path Finder

Hi @renjith.nair

I will try the above query to build the alert configuration , Will configure it to run at every 60min (if results>0)
consider my scenario If the event starts at exacatly 61min before and completes after 10min i.e my alert will find only one event (completed single event) right ,then I may see the only one event corresponding to that event in the past 60min (count>2)

To avoid this alert should run at every 3 hours and should check if there is any job running from more than 2 hours would be fine , I think

Please correct me on this .

0 Karma

raj_mpl
Path Finder

Hi @renjith.nair

The count>2 condition is picking up the single event which is the completed event at 14:28:09 (ends with sting in it)
But that particular event was started at 11:20:00 ( starts with string in it)

In which it is not giving the output correctly

0 Karma

raj_mpl
Path Finder

Will the below one work?

index=myindex ("started plan instance") OR ("successfully completed Plan")
    |"all your rex to extract Job_Name,id  and other fields"|streamstats count,earliest(_time) as _time by id,job_Name
 |where count<2|eval duration=round((now()-_time)/3600,2)|search NOT "successfully completed load" |where duration>1|table job_Name id userid duration
0 Karma

renjith_nair
SplunkTrust
SplunkTrust

@raj_mpl , the schedule purely depends on how fast you should be notified. For eg. lets assume you have scheduled the alerts for every 15 minutes.

Job A & B starts at 08:16 AM.
Alert at 8:30 will not alert because duration is not 2 hours for A &B
Alert at 8:45 will not alert because duration is not 2 hours for A &B
Job B completes
Alert at 9:00 will not alert because duration is not 2 hours for A & for B complete string found, so count is 2
Alert at 9:15 will not alert because duration is not 2 hours for A & for B complete string found, so count is 2
Alert at 9:30 will not alert because duration is not 2 hours for A & for B complete string found, so count is 2
Alert at 9:45 will not alert because duration is not 2 hours for A & for B complete string found, so count is 2
Alert at 10:00 will not alert because duration is not 2 hours for A & for B complete string found, so count is 2
Alert at 10:15 will not alert because duration is not 2 hours for A & for B complete string found, so count is 2
Alert at 10:30 will alert for A because duration is 2 hours and alert is 14 mins delayed.

In this place, if you are scheduling it every 5 minutes, then you get an alert at 10:20 (4 mins delay) because at 10.20 now()-starttime is 10.20-08:16 = 2 hours 4 minutes.

Happy Splunking!
0 Karma

raj_mpl
Path Finder

Hi @renjith.nair
Yes got your point. Now will the below query for the alert to run with a time range of 2 hours for every 20 minutes will give the exact output for me ( Whether the below query will function according to my requirement or do I have to do any modifications in the query before making it enable)

index=myindex ("started plan instance") OR ("successfully completed Plan")
     |"all your rex to extract Job_Name,id  and other fields"|streamstats count,earliest(_time) as _time by id,job_Name
  |where count<2|eval duration=round((now()-_time)/3600,2)|search NOT "successfully completed load" |where duration>2|table job_Name id userid duration
0 Karma

raj_mpl
Path Finder

I don't think that I am getting the required output 🙂

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

why do you think so ? and whats missing in my original search? An example of not getting your required output might help

Happy Splunking!
0 Karma

raj_mpl
Path Finder

Your original search

 index=myindex ("started plan instance") OR ("successfully completed Plan")
        |"all your rex to extract Job_Name,id  and other fields"
        |stats count,earliest(_time) as _time by id,Job_Name
        |where count <2|eval duration=round((now()-_time)/3600,2)|where duration>2

Is free to catch the single events which are having the string "successfully completed" .Here I am running the query for 2 hours time range all the events which are started at before 2 hours but not completed started string in them ,will generate the completed transaction events with same id and Job_name after 15min (exactly when I started the search) .

To avoid that additionally , I had filtered the "successfully Completed" string events.

But when I increase the time range for 24 hours , I am seeing the event duration as 23 ,22,22.3.20,19,14,18...…...2 hours

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

are the id & job_name repeating in your events ? Possible to share some sample events?

Happy Splunking!
0 Karma

raj_mpl
Path Finder

The regex in above query is extracting myfield , id, host fields

0 Karma

skoelpin
SplunkTrust
SplunkTrust

In short, don't use the transaction command. It's slow and requires all data be pushed to the search head which loses the parallelism from the indexers.

https://docs.splunk.com/Documentation/Splunk/7.2.3/SearchReference/Transaction

keeporphans
Syntax: keeporphans=true | false
Description: Specify whether the transaction command should output the results that are not part of any transactions. The results that are passed through as "orphans" are distinguished from transaction events with a _txn_orphan field, which has a value of 1 for orphan results.
Default: false

keepevicted
Syntax: keepevicted=<bool>
Description: Whether to output evicted transactions. Evicted transactions can be distinguished from non-evicted transactions by checking the value of the 'closed_txn' field. The 'closed_txn' field is set to '0', or false, for evicted transactions and '1', or true for non-evicted, or closed, transactions. The 'closed_txn' field is set to '1' if one of the following conditions is met: maxevents, maxpause, maxspan, startswith. For startswith, because the transaction command sees events in reverse time order, it closes a transaction when it satisfies the start condition. If none of these conditions is specified, all transactions are output even though all transactions will have 'closed_txn' set to '0'. A transaction can also be evicted when the memory limitations are reached.
Default: false or 0
0 Karma

raj_mpl
Path Finder

Hi @skoelpin
Thanks for your reply on this and it’s great that the explanation you have given

But my requirement is , I have to create an alert for long running jobs .
( myfield )

Here myfield is a field where all the job names stored

Help me with a query

0 Karma

raj_mpl
Path Finder

Hi @skoelpin

Did you got my point ?

0 Karma

skoelpin
SplunkTrust
SplunkTrust

If you're going to use the transaction command, you should be joining off a single common field rather than 3 of them. Your original query requires each field after the transaction command to have started and complete for it to work. Try limiting it to one field

0 Karma

raj_mpl
Path Finder

Hi @renjith.nair , Could you able to help me in this
I want to create an alert for long running transactions .
Consider events will start with "start" and completes with "closed" string in this with a Customer_Id common in them .

Note : My focus is not on completed transactions , I have to identify the ongoing transactions which are not closed yet

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...