topic Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour? in Splunk Search

How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

shellnight — Wed, 05 Nov 2014 11:26:24 GMT

I need to combine a normal search for 24 hr period with all events and a subsearch on threshold based event where it should query for a certain type of event exceeding a count of 3 in a hour for a host

i ran the below command provided by martin-mueller in earlier thread

https://answers.splunk.com/answers/176574/combining-a-stats-search-and-normal-search.html

index=server earliest=-24h | append [search index=server event-type=high mem-ultilzation | stats count by hostname | where NOT event-type="high mem-utilization" OR count > 3

It does provide the host which exceeded the threshold, but the count provided for the event with the threshold value is incorrect.
It gives the value as 1 with a flat sparkline, when there were 5 actually occurrences in an hour

I need the count to be displayed as 5 and not as 1

Can someone please help in martin's absence

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

MuS — Wed, 05 Nov 2014 11:47:04 GMT

Hi shellnight,

if you look at this run everywhere command:

index=_internal earliest=-24h source="*metrics.log" | bucket _time span=10min | stats count(eval(max(kb) >= 200)) AS myCount by _time, series, host, kb | where myCount > 6 AND NOT series="summary"

does this provide a result you expect?

The search runs over the last 24 hours, builds _time buckets of 10 minutes, counts how many times a series had more then 200 kb throughput per 10 minutes, filters out series="summary" and also results which have less than a count of 6 (6 times a 10min bucket makes up one hour).

Update:
Try this, maybe you have to adapt the field names..but, this will point you towards the solution. I don't know if this is a copy/paste answer.

index=server 
| bucket _time span=1h 
| stats count(eval(event-type="high mem-ultilzation")) AS hi-men-count count(eval(event-type!="high mem-ultilzation")) AS other-count by _time, event-type, host 
| search (event-type="high mem-ultilzation" AND hi-mem-count>="3") OR (NOT event-type="high mem-ultilzation" AND other-count>="0") 
| eval count=if(other-count=="0", hi-mem-count,  other-count) 
| table event-type, host, count

cheers, MuS

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

shellnight — Wed, 05 Nov 2014 15:20:19 GMT

It is an eventtype which occurs frequently on several hosts , I only need the hosts where the event occurs more than 3 times in an hour. All other events in 24hr period need to remain as they are and no conditions need to be applied for then and need the results for both searches in a single table

here is a sample table result that i expect, as you can see only server4 is the only host with highmemutil higher than 3 occurances in an hour.

Eventype hostname Count
Diskspacefull server1 2
highmemutlization server4 5
Networkutlizationhigh server2 20
eventtype5 server3 5

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

MuS — Wed, 05 Nov 2014 17:29:20 GMT

So this Eventype is a field in your events/data and not the eventtype search command from Splunk, right? The Splunk eventtype search command is like a symonym for a search string; for example eventtype=error translates to the search error OR fatal like in the docs http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/eventtypesconf

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

shellnight — Wed, 05 Nov 2014 19:52:24 GMT

Yes you're right.

So is there a way to do what i have requested ?

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

MuS — Wed, 05 Nov 2014 19:55:11 GMT

sure, I'll have a look at it tomorrow....

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

MuS — Thu, 06 Nov 2014 19:33:34 GMT

update ping

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

shellnight — Fri, 07 Nov 2014 10:11:46 GMT

Also MUS , please note that I dont want my existing search which contains filters and macros to be amended , i just want a subsearch to be added to my existing search

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

shellnight — Fri, 07 Nov 2014 10:11:47 GMT

it gives error .
Error in 'eval' command: Typechecking failed. The '==' operator received different types.

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

MuS — Fri, 07 Nov 2014 18:34:01 GMT

look, @martin_mueller did provide a way to go by using a subsearch and I did show you a way without using one. If you don't want to use any of them I suggest you to start here http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial

Re: How to combine a search for all events with a stats subsearch for a certain event that exceeds a threshold within an hour?

shellnight — Sun, 09 Nov 2014 08:48:16 GMT

Yes martin_mueller did provide a way using subsearch and it gives the right information but the event count and sparkline for the threshold event was incorrect .

Though they were 5 events for the host , it came as 1 event with a flat sparkline instead of coming as 5.just need that to be corrected