Need one help to prepare Splunk query.

[dishantgniit]

Hello Team,

I would like to setup Splunk email alert when Log Statement 2 and Log Statement 3 doesn’t execute due to some issues. Log Statement 1 always execute, Log Statement 2 and Log Statement 3 only execute when my scheduler job is working fine. 

Log Statement 2 and Log Statement 3 will not execute and print in log when my scheduler job doesn’t trigger successfully. I would like to get notification when Log Statement 2 and Log Statement 3 not printing in logs after Log Statement 1

I can run the query every 30 min to check if Log Statement 2 and Log Statement 3 printed after Log Statement 1.

I have Splunk Admin access and need help to prepare query. Can you someone please advice.

-------------------------

Log Statement 1: 

“This is Log Line 1 which execute every time when transaction submitted”.

Log Statement 2: 

“This is Log Line 2 will execute after Log Line 1 when transaction in-progress”.

Log Statement 3: 

“This is Log Line 3 will execute after Log Line 2 when transaction completed”.

-------------------------

[bowesmana]

Please share an example of what your log lines look like to help consider the query.

[dishantgniit]

thanks for the response.

We can refer below log statements and keywords for our query - transaction submitted (log 1), transaction in-progress (log 2), transaction completed (log 3)

-------------------------l

Log Statement 1: 

“This is Log Line 1 which execute every time when "transaction submitted"”.

Log Statement 2: 

“This is Log Line 2 will execute after Log Line 1 when "transaction in-progress"”.

Log Statement 3: 

“This is Log Line 3 will execute after Log Line 2 when "transaction completed"”.

-------------------------

[bowesmana]

Can there only ever be a single transaction submitted at any time, so you could not have

transaction submitted (1)
transaction in-progress (1)
transaction submitted (2)
transaction in-progress (2)
transaction completed (2)
transaction completed (1)

another issue is how do you want to handle a transaction that starts inside your 30 minute window but has not yet finished, so has no log2 or log3?

you can do something simple like this

<your search>
| stats 
  sum(eval(if(match(type,"transaction submitted"), 1, 0))) as Submitted
  sum(eval(if(match(type,"transaction in-progress"), 1, 0))) as InProgress
  sum(eval(if(match(type,"transaction completed"), 1, 0))) as Completed
| where !(Submitted=InProgress AND Submitted=Completed)

where the field 'type' contains your log type.

Or you can use the 'transaction' command with 

| transaction startswith="submitted" endswith="completed" keeporphans=t
| where _txn_orphan=1 OR eventcount<3

and this will return only those transactions that do not have submitted/completed and also do not have 3 events in total.

Note that using transaction is not the best approach, as you need to consider your data size, duration of a typical transaction and other things that may affect memory usage, as you can see random results if memory becomes an issue.