topic Re: Transaction command not working with events spanning more than 2 days in Splunk Search

Transaction command not working with events spanning more than 2 days

nabeel652 — Sun, 12 Jun 2016 22:49:46 GMT

Hi,

We have data coming from database showing the status of Orchestrator tasks. Every tasks starts with "In Progress" and ends with "Completed". It can be in "Stuck" state for any period of time (more than 4 days in some cases). So a transaction is a task ID starting with "In Progress" and ending with "Completed" while may remain in "Stuck" State for any period of time.

index="orchestrator_tasks" | transaction ID startswith="In Progress" endswith="Completed" | where Status != "Complete"

Pretty straight forward and works fine but the problem starts when there are transactions that span over few days. I can see Splunk bundles them in Transactions "In Progress" -> "Stuck" And Then "Stuck" -> "Completed" if startswith and endswith conditions are removed. In presence of these conditions it keeps showing the first part i. e "In Progress" -> "Stuck" therefore showing such tasks as Stuck whereas they've been Completed.

My thought is that the problem is with maximum time span Splunk looks for completed event.

Re: Transaction command not working with events spanning more than 2 days

Raghav2384 — Mon, 13 Jun 2016 03:23:27 GMT

maxspan is to -1 unless specified. In your case, i do not think maxspan is the issue

maxspan
Syntax: maxspan=[s | m | h | d]
Description: Specifies the maximum length of time in seconds, minutes, hours, or days that the events can span. The events in the transaction must span less than integer specified for maxspan. If the value is negative, maxspan is disabled and there is no limit.
Default: -1 (no limit)

Try adding maxevents=-1 and run the same search and see if it picks the multiple day spanning events. Since you are tracing one transaction spanning over multiple days, i doubt if the event limits is the cause. Is it possible to post a sample event set that's spanned over days? What if you hard code few IDs in the main search and see if transaction is able to catch it? Hope this leads you the right way.

Example: index="orchestrator_tasks" ID="abcdef" OR ID="ghijkl" OR ID="mnopqr"|transaction ID startswith="In Progress" endswith="Completed" maxevents=-1|where Status != "Complete"

Thanks,
Raghav

Re: Transaction command not working with events spanning more than 2 days

nabeel652 — Mon, 13 Jun 2016 03:26:21 GMT

Cool. I already resolved it with maxevents=-1 and was about to update my question but your reply came through. Anyway, its correct answer!

Re: Transaction command not working with events spanning more than 2 days

Raghav2384 — Mon, 13 Jun 2016 03:31:09 GMT

Excellent...glad it worked out...please accept this as answer so that it could help folks catch it easily 🙂

Thanks,
Raghav

Re: Transaction command not working with events spanning more than 2 days

MuS — Mon, 13 Jun 2016 03:57:00 GMT

@Raghav2384, please post it as answer so it can be accepted 😉 I converted your comment now.

Re: Transaction command not working with events spanning more than 2 days

Raghav2384 — Mon, 13 Jun 2016 04:00:16 GMT

Thanks Michael , @MuS 🙂