Efficient use of transaction or eventstats command

[ash2l]

Hello,

I am trying to find following
1) Events that starts with WSQ0001 and ends with AAA9999.
2) Find EVE_INCIDENT_LOCATOR_NM from those events and match it up with other events to find State value, because the above 2 event do not carry that field.
3) Find the latest events for AAA9999
4) create a table with month, state, error description and count for each unique errors

I am using following query and my user dispatch directory limit is 1GB which gets filled up with 2 hours of search data. The request is to run this search for 1 month of data. How do I achieve it? Any other commands that I can use which consumes very less size? I was using transaction command but it was not scalable at all, so I ended up using eventstats but I found same issues with that as well.

index="ibb" host="xx" OR host="yy"  | eval Month_Value = strftime(_time, "%m-%Y") | rex max_match=100 "Rejected From: (?<reject_from>.*?)&amp;lt;br" 
| eventstats values(State_Value) as State_Value, values(AppName_Value) as AppName_Value_list by EVE_INCIDENT_LOCATOR_NM | search AppName_Value_list="*AAA9999*" AND AppName_Value_list="*WSQ0001*" | search AppName_Value="*AAA9999*" | dedup EVE_INCIDENT_LOCATOR_NM  | rex mode=sed  field=reject_from "s/&amp;lt;/ /g" | rex mode=sed  field=reject_from "s/&amp;gt;/ /g" | stats count as Total_Events, values(reject_from) as Description by State_Value,Month_Value, EVE_INCIDENT_LOCATOR_NM | mvexpand Description | eval Line_10_Description = if(Description like "%/010%",Description,"") | eval Line_19_Description = if(Description like "%/019%",Description,"")  | stats  max(Total_Events) as Total_Events, values(Description) as Description, values(Line_10_Description) as Line_10_Description, values(Line_19_Description) as Line_19_Description by State_Value, Month_Value, EVE_INCIDENT_LOCATOR_NM  | mvcombine delim="," Description | stats count as Total_Events, values(Line_10_Description) as Line_10_Description, values(Line_19_Description) as Line_19_Description by State_Value, Month_Value, Description | table Month_Value, State_Value, Line_10_Description, Line_19_Description, Total_Events 

[maciep]

Without seeing and knowing your data, it might be hard for us to give you good advice. We'll say "try this" and you'll probably say "that won't work because...". But nonetheless, here are some thoughts:

It looks like you do most of your processing after you filter down to just the AAA9999 events. But can we flip that approach? Can you perform that logic at the event level. And then use just stats to do the filtering. For example, what about determining if an event is an AAA9999. Then set the Description fields accordingly - nothing if not that event or what they should be otherwise.

And when you do your stats, then the description list should already be what you need it to be. And you can also count the number of AAA9999 events specifically. From there, you can just filter on AppNames you're interested in and shouldn't have to do much more processing.

Something like this maybe:

index="ibb" (host="xx" OR host="yy") 
| eval is_AAA9999 = if(match(AppName_Value,"AAA9999"),1,0)
| eval Month_Value = strftime(_time, "%m-%Y") 
| rex max_match=100 "Rejected From: (?<reject_from>.*?)&amp;lt;br" 
| rex mode=sed  field=reject_from "s/&amp;lt;/ /g" 
| rex mode=sed  field=reject_from "s/&amp;gt;/ /g" 
| eval Description = if(is_AAA9999=1,reject_from,"")
| eval Line_10_Description = if(Description like "%/010%",Description,"") 
| eval Line_19_Description = if(Description like "%/019%",Description,"")  
| stats values(Description) as Description, values(Line_10_Description) as Line_10_Description, values(Line_19_Description) as Line_19_Description, values(State_Value) as State_Value, values(AppName_Value) as AppName_Value_list, count(eval(is_AAA999=1)) as total_events by EVE_INCIDENT_LOCATOR_NM 
 | search AppName_Value_list="*AAA9999*" AND AppName_Value_list="*WSQ0001*" 

Again, this may not work exactly with your data, but hopefully it will give you some ideas. Sometimes thinking in order of how we would do things isn't always the best order for how Splunk should do them.

Also, if you have to do this on a regular basis, maybe run a search on a regular basis to create a lookup of the EVE_INCIDENT_LOCATOR_NM values that meet the criteria - especially if this search is going to run over a previous timeframe, like last month.

And of course, you could try doing something similar with a subsearch, but not sure how efficient that would be either.

[martin_mueller]

Without knowing your data, I'd try replacing eventstats with stats.

https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo...