topic Transaction and Filldown from different hosts in Splunk Search

Transaction and Filldown from different hosts

tbrown — Thu, 13 Aug 2020 20:11:39 GMT

I have the following query

Where I look at all events and then use rex and eval to evaluate the necessary rising and falling nodes of each transaction. Then I sort +_time so that I can filldown those fields to all the events, then use | where to filter out any events that aren't coming from either the rising or falling nodes, and then finally sort-_time so that I can form the transaction.

This works perfectly for all transactions where the start and end events are coming from hosts that are the Rising_Node or the Falling_Node.

However, some of my events that I want to turn into transactions aren't like this. They have the same necessary start and end events, but those start and end events are NOT on the Rising_Node or the Falling_Node. Therefore, my query does not work because it the " | where (host==Rising_Node OR host==Falling_Node)" always filters out those start and end events so the transaction cannot occur.

Does anyone have a workaround for these peculiar events where the start and end events are not on the rising or falling nodes?

Here is what the scambled data looks like for some of the transactions that have start events that are not on the rising or falling nodes:

2020/08/11 11:40:18.473, NOT_RISING/FALLING_NODE , Requested Falling node 'Falling_Node' by user '....'" (Start Event) 2020/08/11 11:40:44.512, Falling_Node, , <....information....> 2020/08/11 11:40:45.512, Rising_Node, , <....information....> 2020/08/11 11:40:49.512, NOT_RISING/FALLING_NODE, , <....information....> 2020/08/11 11:40:49.889, NOT_RISING/FALLING_NODE, , <....information....> 2020/08/11 11:40:50.512, Rising_Node, , <....information....> 2020/08/11 11:40:51.512, Rising_Node, , <....information....> 2020/08/11 11:40:55.889, NOT_RISING/FALLING_NODE, , <....information....> 2020/08/11 11:40:59.512, Rising_Node, , <....information....> 2020/08/11 11:40:59.889, NOT_RISING/FALLING_NODE, , <....information....> 2020/08/11 11:41:13.915, NOT_RISING/FALLING_NODE, Completed Transaction (End Event)

This is a sample of events in which my query doesn't work correctly because the Start and End Events are NOT the Rising or Falling Node. However, I want to filter out all the NOT_RISING/FALLING_NODE events within the transaction.

Re: Transaction and Filldown from different hosts

isoutamo — Thu, 13 Aug 2020 19:47:26 GMT

can you share scrambled data so the community could help you?

r. Ismo

Re: Transaction and Filldown from different hosts

tbrown — Thu, 13 Aug 2020 20:12:01 GMT

I've edited my original post to include a sample of events.

Re: Transaction and Filldown from different hosts

isoutamo — Thu, 13 Aug 2020 20:21:54 GMT

If I understood you want also 1st and last line of your example to be in transaction with all those lines from Rising or Falling nodes?

Re: Transaction and Filldown from different hosts

tbrown — Thu, 13 Aug 2020 20:25:44 GMT

Yes, I need those events in the transaction as they are the constant start and end events I base the transaction on. The issue is that they aren't always necessarily coming from the Rising Or Falling Node hosts.

Re: Transaction and Filldown from different hosts

isoutamo — Thu, 13 Aug 2020 20:39:53 GMT

How about this:

Before filtering events out add new columns which is populated with some know values if event is from Falling, Rising nodes or it contains those start/end transaction info? After that you actually can avoid the use of transaction command as you can replace it with stats .... by <your additional column>.

r. Ismo

Re: Transaction and Filldown from different hosts

tbrown — Thu, 13 Aug 2020 20:53:17 GMT

Do you have an example of how I could do it? I'm having a little trouble trying to actually get this to work in the query.

Re: Transaction and Filldown from different hosts

isoutamo — Thu, 13 Aug 2020 21:38:48 GMT

Something like this:

... | rex +"(?<trc_end>Completed Transaction)" | eval belongs_to_trc = if( host==Falling_Node) OR host==Rising_Node OR isnotnull(Falling_Node) OR isnotnull(trc_end), 1, 0) ... | where belongs_to_trc == 1

r. Ismo

Re: Transaction and Filldown from different hosts

tbrown — Fri, 14 Aug 2020 19:13:53 GMT

Just wondering, how would that filter out any events after the End Event? Because I still have events coming in after the end event that are from the rising and falling nodes, but I don't want to include those because they come after the end event.

Re: Transaction and Filldown from different hosts

isoutamo — Fri, 14 Aug 2020 21:18:43 GMT

Transaction is the only (easy and reasonable) way to achieve this.

Re: Transaction and Filldown from different hosts

tbrown — Fri, 14 Aug 2020 21:25:10 GMT

So is there no solution? Using transaction would bring me back to the original problem I posted.

Re: Transaction and Filldown from different hosts

isoutamo — Fri, 14 Aug 2020 22:03:21 GMT

I think that it now doable as you could use earlier added field belongs to transactions as unique field and then start and end as earlier? If I have understood right this should work.

r. Ismo