Splunk Search
Highlighted

Multisearch alert to go off only from one side + sourcetype count optimisation

Path Finder

I've tried to set up an alert to go off whenever the number of hosts from one search is not the same for another search, but I only want it to go off from one side (so if the number of hosts in search A < the number of hosts in search B, it should go off but if the number of hosts in search A >= the number of hosts in search B, I don't want it to go off). As of late, I've seen the number of alerts increase substantially but then when I check the individual searches, I can see it's the latter issue where search A hosts exceed search B hosts - how can I fix this so it only alerts from one side?

| set diff
[search index=_internal source=.../metrics.log "..." | dedup host | sort host | table host ] << Search A
[search index=* sourcetype=core-server-event-tracking-api | dedup host | sort host | table host ] << Search B
| rename host as "Missing Host(s)"

Also, is there a better way of counting the number of unique hosts from a sourcetype, e.g. core-server-event-tracking-api, rather than counting across all sources?

0 Karma
Highlighted

Re: Multisearch alert to go off only from one side + sourcetype count optimisation

SplunkTrust
SplunkTrust

hello there,
i believe there are plenty of ways to do that but here is my clumsy version:

 index="_internal" source=*metrics.log* 
    | bin span=5m _time 
    | stats dc(host) as unique_hosts_1 by _time
    | appendcols [search index =*  sourcetype=core-server-event-tracking-api
    | bin span=5m _time 
    | stats dc(host) as unique_host_2 by _time]
    | table _time unique*
    | where unique_hosts_1  > unique_hosts_2

save as an alert if count is equal or greater than 1
used the stats dc (distinct count) to check how many unique hosts are in each search
hope it helps

0 Karma
Highlighted

Re: Multisearch alert to go off only from one side + sourcetype count optimisation

Path Finder

Hi Adonio,

I tried using your method and it didn't work unfortunately. I initially thought it was because I didn't add the actual string I was searching for in the metrics log, but I couldn't get it to work after adding that.

I actually asked this question in a different way and I got the answer I wanted from there:
https://answers.splunk.com/answers/560584/using-set-diff-to-compare-searches-but-outputting.html

To summarise, I had to use a multisearch to get both sets of results and then it's suggested to use mvcount and where to display what I was initially looking for:

| multisearch
[...Search 1...]
[...Search 2...]
| fields host sourcetype
| eval host=upper(host)
| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2

0 Karma
Highlighted

Re: Multisearch alert to go off only from one side + sourcetype count optimisation

Path Finder

I used the following in the end:

| multisearch
[search index=* sourcetype=x... ]
[search index=* sourcetype=y... ]
| fields host sourcetype
| eval host=upper(host)| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2 AND sourcetype=x

I realised that there was an issue on the boxes themselves so once I fixed the inputs.conf file and restarted the agent, it was picking up as normal so I was able to remove the "AND sourcetype=x"

View solution in original post

0 Karma