I've tried to set up an alert to go off whenever the number of hosts from one search is not the same for another search, but I only want it to go off from one side (so if the number of hosts in search A < the number of hosts in search B, it should go off but if the number of hosts in search A >= the number of hosts in search B, I don't want it to go off). As of late, I've seen the number of alerts increase substantially but then when I check the individual searches, I can see it's the latter issue where search A hosts exceed search B hosts - how can I fix this so it only alerts from one side?
| set diff
[search index=_internal source=.../metrics.log "..." | dedup host | sort host | table host ] << Search A
[search index=* sourcetype=core-server-event-tracking-api | dedup host | sort host | table host ] << Search B
| rename host as "Missing Host(s)"
Also, is there a better way of counting the number of unique hosts from a sourcetype, e.g. core-server-event-tracking-api, rather than counting across all sources?
i believe there are plenty of ways to do that but here is my clumsy version:
index="_internal" source=*metrics.log* | bin span=5m _time | stats dc(host) as unique_hosts_1 by _time | appendcols [search index =* sourcetype=core-server-event-tracking-api | bin span=5m _time | stats dc(host) as unique_host_2 by _time] | table _time unique* | where unique_hosts_1 > unique_hosts_2
save as an alert if count is equal or greater than 1
used the stats dc (distinct count) to check how many unique hosts are in each search
hope it helps
I tried using your method and it didn't work unfortunately. I initially thought it was because I didn't add the actual string I was searching for in the metrics log, but I couldn't get it to work after adding that.
I actually asked this question in a different way and I got the answer I wanted from there:
To summarise, I had to use a multisearch to get both sets of results and then it's suggested to use mvcount and where to display what I was initially looking for:
| fields host sourcetype
| eval host=upper(host)
| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2
I used the following in the end:
[search index=* sourcetype=x... ]
[search index=* sourcetype=y... ]
| fields host sourcetype
| eval host=upper(host)| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2 AND sourcetype=x
I realised that there was an issue on the boxes themselves so once I fixed the inputs.conf file and restarted the agent, it was picking up as normal so I was able to remove the "AND sourcetype=x"