Splunk Search

Multisearch alert to go off only from one side + sourcetype count optimisation

sepkarimpour
Path Finder

I've tried to set up an alert to go off whenever the number of hosts from one search is not the same for another search, but I only want it to go off from one side (so if the number of hosts in search A < the number of hosts in search B, it should go off but if the number of hosts in search A >= the number of hosts in search B, I don't want it to go off). As of late, I've seen the number of alerts increase substantially but then when I check the individual searches, I can see it's the latter issue where search A hosts exceed search B hosts - how can I fix this so it only alerts from one side?

| set diff
[search index=_internal source=.../metrics.log "..." | dedup host | sort host | table host ] << Search A
[search index=* sourcetype=core-server-event-tracking-api | dedup host | sort host | table host ] << Search B
| rename host as "Missing Host(s)"

Also, is there a better way of counting the number of unique hosts from a sourcetype, e.g. core-server-event-tracking-api, rather than counting across all sources?

0 Karma
1 Solution

sepkarimpour
Path Finder

I used the following in the end:

| multisearch
[search index=* sourcetype=x... ]
[search index=* sourcetype=y... ]
| fields host sourcetype
| eval host=upper(host)| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2 AND sourcetype=x

I realised that there was an issue on the boxes themselves so once I fixed the inputs.conf file and restarted the agent, it was picking up as normal so I was able to remove the "AND sourcetype=x"

View solution in original post

0 Karma

sepkarimpour
Path Finder

I used the following in the end:

| multisearch
[search index=* sourcetype=x... ]
[search index=* sourcetype=y... ]
| fields host sourcetype
| eval host=upper(host)| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2 AND sourcetype=x

I realised that there was an issue on the boxes themselves so once I fixed the inputs.conf file and restarted the agent, it was picking up as normal so I was able to remove the "AND sourcetype=x"

0 Karma

adonio
Ultra Champion

hello there,
i believe there are plenty of ways to do that but here is my clumsy version:

 index="_internal" source=*metrics.log* 
    | bin span=5m _time 
    | stats dc(host) as unique_hosts_1 by _time
    | appendcols [search index =*  sourcetype=core-server-event-tracking-api
    | bin span=5m _time 
    | stats dc(host) as unique_host_2 by _time]
    | table _time unique*
    | where unique_hosts_1  > unique_hosts_2

save as an alert if count is equal or greater than 1
used the stats dc (distinct count) to check how many unique hosts are in each search
hope it helps

0 Karma

sepkarimpour
Path Finder

Hi Adonio,

I tried using your method and it didn't work unfortunately. I initially thought it was because I didn't add the actual string I was searching for in the metrics log, but I couldn't get it to work after adding that.

I actually asked this question in a different way and I got the answer I wanted from there:
https://answers.splunk.com/answers/560584/using-set-diff-to-compare-searches-but-outputting.html

To summarise, I had to use a multisearch to get both sets of results and then it's suggested to use mvcount and where to display what I was initially looking for:

| multisearch
[...Search 1...]
[...Search 2...]
| fields host sourcetype
| eval host=upper(host)
| stats values(sourcetype) as sourcetype by host
| where mvcount(sourcetype)<2

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...