topic Transaction with start & end with the same time in _raw ? in Splunk Search

Transaction with start & end with the same time in _raw ?

sjringo — Wed, 21 Feb 2024 15:37:43 GMT

I am trying to create a Transaction where my starting and ending 'event' have exactly the same time.

In _raw the time is "Wed Feb 21 08:15:01 CST 2024"

My current SPL is: | transaction keeporphans=true host aJobName startswith=("START of script") endswith=("COMPLETED OK" OR "ABORTED, exiting with status")

But my transaction only has the starting event. So I added the following which had no change ?

| eval _time = case(
match(_raw, "COMPLETED OK"), _time +5,
match(_raw, "ABORTED"), _time +5,
true(),_time)
| sort _time

| transaction keeporphans=true host aJobName startswith=("START of script") endswith=("COMPLETED OK" OR "ABORTED, exiting with status")

When I added the above changes, when I look the the events in the 'Time' columns they are 5 seconds apart, yet Tranaction does not associate them ?

2/21/24 8:15:01.000 AM (Starting Event)

2/21/24 8:15:06.000 AM (Ending Event)

Re: Transaction with start & end with the same time in _raw ?

richgalloway — Wed, 21 Feb 2024 20:47:29 GMT

Have you tried using the maxspan option to limit how far apart the startswith and endswith events can be?

| transaction keeporphans=true host aJobName startswith=("START of script") endswith=("COMPLETED OK" OR "ABORTED, exiting with status") maxspan=0

Re: Transaction with start & end with the same time in _raw ?

sjringo — Wed, 21 Feb 2024 21:05:32 GMT

Putting maxspan option does work for the one particular event where the start/stop events happen at the same time.

The next issue that comes up is that there are around 80 "transactions" that I am monitoring that can have a duration of over an hour.

The only way I can think of making this work is to have two different transaction creation lines that are inside of a case statement? One with the maxspan and one without depending upon a job name that I am extracting earlier in my code...

Is that possible or do you have any other ideas/suggestions ?

Re: Transaction with start & end with the same time in _raw ?

richgalloway — Wed, 21 Feb 2024 21:31:27 GMT

Eighty transactions of up to an hour is a new requirement that my previous suggestion will not handle.

The transaction command is pretty inefficient and will become less so when it has to track many transactions over a long time range.

Rather than help you with a specific, sub-optimal solution, let's see if there's another solution to the problem. What problem are you trying to solve?

Re: Transaction with start & end with the same time in _raw ?

sjringo — Wed, 21 Feb 2024 22:02:58 GMT

Sure.. So, here it goes..
I have a dashboard that is tracking 'jobs'... Completed jobs and this particular widget is tracking 'running' jobs (start but no end). I might be tracking around 80 jobs but there should not be more than 5 or 6 'running' at any particular time. So, not creating 80 transactions.

Everything is working as designed but this one job that starts and ends at the same time showed up in my 'running' jobs widget and then is missing from my completed jobs widget.

Once I run my initial 'search' for log events here is what im doing.

index=anIndex sourcetype=aSourcetype (aJob1 OR aJob2 OR aJob3) AND ("START of script" OR "COMPLETED OK" OR "ABORTED, exiting with status" ) | rex field=_raw "Batch::(?<aJobName>[^\s]*)" | transaction keeporphans=true host aJobName startswith=("START of script") endswith=("COMPLETED OK" OR "ABORTED, exiting with status") | eval closed_txn = if ( isnull(closed_txn),0,closed_txn) | search closed_txn=0 | sort _time | eval aDay = strftime(_time, "%a. %b. %e, %Y") | eval aStartTime=strftime(_time, "%H:%M:%S %p") | eval aDuration=tostring((now()-_time), "duration") | eval aEndTime = "--- Running ---" | table aHostName aDay aJobName aStartTime aEndTime aDuration

But, this one job is causing me issues as Transaction is not picking up the start/end that have the same _time

Re: Transaction with start & end with the same time in _raw ?

richgalloway — Sat, 24 Feb 2024 21:23:55 GMT

Here's a method that often works. Search for Start and Completed events, keeping only the most recent for each host and job. Then discard all of the Completed events. What's left will be a list of uncompleted jobs.

This approach will fail if the Start and Complete events are at the exact time and in the wrong order.

index=anIndex sourcetype=aSourcetype (aJob1 OR aJob2 OR aJob3) AND ("START of script" OR "COMPLETED OK" OR "ABORTED, exiting with status" ) | dedup host aJobName | search "START of script" | rex field=_raw "Batch::(?<aJobName>[^\s]*)" | sort _time | eval aDay = strftime(_time, "%a. %b. %e, %Y") | eval aStartTime=strftime(_time, "%H:%M:%S %p") | eval aDuration=tostring((now()-_time), "duration") | eval aEndTime = "--- Running ---" | table aHostName aDay aJobName aStartTime aEndTime aDuration

Re: Transaction with start & end with the same time in _raw ?

sjringo — Sun, 25 Feb 2024 00:48:17 GMT

I was stuck on trying to get Transaction to work 😉

It was on my list of things to do, to write it similar to the way you did but had not had the time to get to it.

I ran a few tests and appears to solve the 'issue'

I dont know the specifics but I guess trying to 'alter' _time really does not change the underlying value ?

Re: Transaction with start & end with the same time in _raw ?

richgalloway — Sun, 25 Feb 2024 01:04:54 GMT

You can change _time (or any field) in a query, but it doesn't change the indexed data (nothing does).