Multisearch alert to go off only from one side + sourcetype count optimisation

[sepkarimpour]

I've tried to set up an alert to go off whenever the number of hosts from one search is not the same for another search, but I only want it to go off from one side (so if the number of hosts in search A < the number of hosts in search B, it should go off but if the number of hosts in search A >= the number of hosts in search B, I don't want it to go off). As of late, I've seen the number of alerts increase substantially but then when I check the individual searches, I can see it's the latter issue where search A hosts exceed search B hosts - how can I fix this so it only alerts from one side?

| set diff
[search index=_internal source=.../metrics.log "..." | dedup host | sort host | table host ] << Search A
[search index=* sourcetype=core-server-event-tracking-api | dedup host | sort host | table host ] << Search B
| rename host as "Missing Host(s)"

Also, is there a better way of counting the number of unique hosts from a sourcetype, e.g. core-server-event-tracking-api, rather than counting across all sources?

[sepkarimpour · Accepted solution]

I used the following in the end:

| multisearch
[search index=* sourcetype=x... ]
[search index=* sourcetype=y... ]
| fields host sourcetype
| eval host=upper(host)| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2 AND sourcetype=x

I realised that there was an issue on the boxes themselves so once I fixed the inputs.conf file and restarted the agent, it was picking up as normal so I was able to remove the "AND sourcetype=x"

[adonio]

hello there,
i believe there are plenty of ways to do that but here is my clumsy version:

 index="_internal" source=*metrics.log* 
    | bin span=5m _time 
    | stats dc(host) as unique_hosts_1 by _time
    | appendcols [search index =*  sourcetype=core-server-event-tracking-api
    | bin span=5m _time 
    | stats dc(host) as unique_host_2 by _time]
    | table _time unique*
    | where unique_hosts_1  > unique_hosts_2

save as an alert if count is equal or greater than 1
used the stats dc (distinct count) to check how many unique hosts are in each search
hope it helps

[sepkarimpour]

Hi Adonio,

I tried using your method and it didn't work unfortunately. I initially thought it was because I didn't add the actual string I was searching for in the metrics log, but I couldn't get it to work after adding that.

I actually asked this question in a different way and I got the answer I wanted from there:
https://answers.splunk.com/answers/560584/using-set-diff-to-compare-searches-but-outputting.html

To summarise, I had to use a multisearch to get both sets of results and then it's suggested to use mvcount and where to display what I was initially looking for:

| multisearch
[...Search 1...]
[...Search 2...]
| fields host sourcetype
| eval host=upper(host)
| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2