Why does the Transaction command for "All Time" not include all recent events?

[wjblazek]

I have search lots of transaction questions and don't see any related to this question.

I have a search that defines a common field across multiple events.
This search displays lots of events consistently back to the 1st relevent event up through the current time for all time periods:

• When it runs for the Last 24 hours time period, it lists 3838 events.

• When it runs for the Last 7 Days time period, it lists 24,027 events from 9/5 – 9/12/14, including today’s events.

• When it runs for the Week-to-Day time period, it lists 19,274 events from 9/7 – 9/12/14, including today’s events.

• When it runs for the Month-to-Day time period, it lists 48,192 events from 9/1 – 9/12/14, including today’s events.

• When it runs for the Last 30 Days time period, it lists 79,311 events from 8/22 – 9/12/14, including today’s events.

• When it runs for the Previous Month, August, time period, it lists 31,156 events from 8/22 – 8/31/14, nothing before 8/22.

• When it runs for the Year-to-Day time period, it lists 79,311+ events from 8/22 – 9/12/14, including today’s events.

Take the same search and add the transaction command on that common field to create
transactions that combines events with matching values of that common field:

" ... | rex field=_raw "(?P CCID...) ..." | transaction CCID

(The actual search doesn't display correctly here. The <> around the CCID don't display here but they are there.)

• When it runs for the Last 24 hours time period, it shows 3605 events since yesterday at this time.

• When it runs for the Yesterday time period, it shows 4883 events yesterday.

• When it runs for the Week-to-date time period, it only shows data for the 1st 2 days of the Week-to-date period, nothing for yesterday or the Last 24 Hours.

• The previous 2 commands demonstrate that there is data from yesterday and today which don’t show up in the Week-to-date period.

• When it runs for the Month-to-Date time period, it only shows data for the 1st 2 days of the month and none of the data shown by the previous 3 commands, Week-to-Date, Yesterday and Last 24 Hours.

• When it runs for the Year-to-Date or All Time time periods, it only shows data through August, none of the data shown for the previous 4 commands.

Adding the transaction command appears to not include all of the more recent data for several time periods:

Month-to-Date, Year-to-Date or All Time time periods,
that is display by the last 24 Hour, Yesterday commands and the searches without the transaction command.

This occures in both regular search windows and in dashboard panels.

Is there a reason for this?

It appears to not be correct.

Thanks much for your help.

[wpreston · Accepted solution]

Transaction had a limit of 10,000 results by default in prior versions of Splunk. I don't know if that is still the default in Splunk 6.x or if it has been increased/changed/removed as it is hard to tell from the documentation. If you look at the last line of the docs for keepevicted, you'll see it say that "a transaction can also be evicted when the memory limitations are reached. " So some of your results could be evicted due to memory management, but this can be overcome by using the keepevicted flag. Try your search with the keepevicted flag set to 't' and see if that makes a difference. You might also add in criteria for startswith and endswith (if possible) to increase transaction's performance.

... | rex field=_raw "(?P CCID...) ..." | transaction CCID startswith=<starting criteria> endswith=<ending criteria> keepevicted=t

[martin_mueller]

If you know for certain that a transaction only has two events then specifying that makes life a lot easier for Splunk - it doesn't need to keep a transaction in memory once two events have been found!

Do check though if you can specify startswith and endswith - often those are more robust in case your assumption of "only two events per transaction" doesn't hold for some weird corner cases.

[wjblazek]

Thank you all very much for the fast replies!

Keepevicted=t is the primary solution for our scenario and works perfectly.
I've also added the maxevents=2 flag since we only have 2 events per transaction.

The CCID can be either short or long lived value in real-time but
we are looking at this data in log files which are appended to but never deleted so
all our data is effectively static.

I'm sure we will find many more uses for transaction that may need more of the options/flags.

[gkanapathy]

On the kind of right track, but you also need to pay attention to the maxevents, maxopenevents, and maxopentxn parameters, and not just keepevicted

[martin_mueller]

How is each value of CCID distributed over time? For how long does each value "live"?
For example, is it a short-lived globally-unique request ID type of thing, or a longer-living thing such as a contract ID?