topic Re: Transaction command is not closing transactions and is hitting the max open transaction limit (maxopentxn). in Splunk Search

Transaction command is not closing transactions and is hitting the max open transaction limit (maxopentxn).

andrewcg — Tue, 26 May 2020 17:00:26 GMT

We recently upgraded to from 7.1.2 to 8.0.3 on on-prem Splunk Enterprise. A previously working saved search is no longer returning the correct results.

| transaction session_id maxspan=30s

Looking into it looks like the transaction command is no longer closing connections when the maxspan (30s) value is hit. This leaves all transactions open and then the search ends when it hits the default of 5000.

I need to create transactions out of 650000 entries (two or three lines each), so needless to say this search no longer functions. I can confirm this behavior, by:

| stats count by closed_txn shows all the transactions returned as closed_txn=0
adding maxopentxn=5500 to the transaction command causes the number of returned results to go from 5000 to 5500
adding maxevents=2 only closes some of the events

closed_txn, eventcount, count

0          1           1041
0          2           4458
1          2           1654

Transactions are supposed to close when:

The 'closed_txn' field is set to '1' if one of the following conditions is met: maxevents, maxpause, maxspan, startswith.

https://docs.splunk.com/Documentation/Splunk/8.0.3/SearchReference/Transaction -> Memory control options -> keepevicted

Re: Transaction command is not closing transactions and is hitting the max open transaction limit (maxopentxn).

niketn — Wed, 30 Sep 2020 05:32:23 GMT

@andrewcg which are the fields you want to stitch together after transaction command?

You should ideally try stats for your correlation instead of transaction. There are several examples of stats for transaction on Splunk Answers.

<yourCurrentSearch>
| fields _time, session_id, field1, field2, field3
| eval Time=_time
| bin _time span=30s
| stats count as eventcount list(*) as list_* by _time session_id
| eval maxTime=mvindex(list_Time,0),minTime=mvindex(list_Time,eventcount-1), duration=maxTime-minTime
| fields - minTime maxTime

Following is a runanywhere example based on Splunk's _internal index just to generate some data and correlate using component (similar to session_id)

index=_internal sourcetype=splunkd 
| fields _time, component, log_level, date_hour, date_minute, date_second
| eval Time=_time
| bin _time span=30s 
| stats count as eventcount list(*) as list_* by component _time
| eval maxTime=mvindex(list_Time,0),minTime=mvindex(list_Time,eventcount-1), duration=maxTime-minTime
| fields - minTime maxTime

Please try out and confirm!

Re: Transaction command is not closing transactions and is hitting the max open transaction limit (maxopentxn).

andrewcg — Wed, 30 Sep 2020 05:33:55 GMT

Worked perfectly for me. I could not picture in my mind how the transaction command could be duplicated with a stats command based on the Splunk documentation, your example illustrated it for me. I now have a faster and more importantly functional search. Thank you.

I was able to drop the _time bin part as the session_id is unique enough to not be duplicated in a day and I was only using the maxspan to close open transactions. My search went from:

transaction session_id maxspan=30s

| stats count as eventcount list(mac) AS mac, list(hostname) AS hostname, list(nas_ip) AS nas_ip, list(nas_port) AS nas_port, list(nas_id) AS nas_id, list(attr_value) AS attr_value, list(profiles) AS profiles by session_id

That allowed me to keep all the rest of the search on either side the same.

Re: Transaction command is not closing transactions and is hitting the max open transaction limit (maxopentxn).

niketn — Wed, 27 May 2020 15:09:30 GMT

I prefer transaction when for scenarios where I need to drilldown with a particular session_id or for more complex scenarios where things like maxpause and other conditions are in place. A good guide to Event Grouping and Correlation is this Splunk Documentation: https://docs.splunk.com/Documentation/Splunk/latest/Search/Abouteventcorrelation

However, there is more to correlation and commands like union, multisearch also needs to be considered under specific circumstances.

Re: Transaction command is not closing transactions and is hitting the max open transaction limit (maxopentxn).

andrewcg — Wed, 30 Sep 2020 15:36:51 GMT

I totally forgot to follow up on this issue, thank you for commenting on this issue. Using `stats` was the recommendation from the Splunk support. Not only does using `stats` result in all the entries being reflected in the results, it is so much faster.