How to create an alert by comparing with previous alert sent to summary index?

[vrmandadi]

I am trying to create an alert and send the alert details to summary index.Below is the search I am using.I have scheduled the below search everyday at 2AM and look for yesterday data and send alert and then send same data to summary index..I am trying to create another alert to compare the data with summary index and send alert only if there is a difference in results..I am trying to compare the combination of host gpu and VBIOS_Version fields..if all these are different then send an alert

Query for alert

index=preo  host IN(*) 
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version |collect index=summary marker="summary_type=test"
| eval details= "preos Splunk: ".host.  " node VBIOS mismatch " .gpu.  " " .VBIOS_Version. " Common:" .Common_VBIOS_Version." date:" .time 
| table details

Below is the query I tried to compare with summary index and send if there is a change 

index=preos host IN(*) *GPU*: PCISLOT*
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| join host gpu VBIOS_Version 
    [search index=summary   summary_type=test 

| table gpu orig_host VBIOS_Version 
| rename orig_host as host ]

[johnhuang]

If you're just comparing yesterday to today's change, you can probably skip using the summary index table.

index=preos host IN(*) *GPU*: PCISLOT* earliest=-2d@d latest=-0d@d
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| eval event_date=strftime(_time,"%Y-%m-%d")
| stats latest(_time) AS _time latest(*) AS * BY event_date host gpu
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time event_date host gpu ios common_ios gpu_ios 
| eval event_time=strftime(_time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| fillnull value=""
| eval record_hash=MD5(host ." | ".gpu." | ".VBIOS_Version)
| eventstats min(_time) AS first_seen BY record_hash
| where first_seen>relative_time(now(), "-1d@d")
| eval first_seen_time=(first_seen,"%Y-%m-%d %H:%M:%S")
| table event_time gpu host VBIOS_Version

[jdunlea]

That will work too.

However if there is a lot of data in the index, then it will mean you have to search double the data by expanding the time range from 1 day to 2 days.  Additionally, it essentially means that you are "double searching" at least one day's worth of data. 

On the other hand, there are also technical constraints to using the subsearch and join method.

Just worth noting the tradeoffs between both options. 

[johnhuang]

@jdunlea, yep agreed, the best approach is scenario based.

There's a mistake in the query which write to the summary index. In particular, the _time field was renamed to time and as a result, the timestamp of data saved to the summary index would no longer be accurate.

Hence, not using the summary index data was the most straightforward approach.

[jdunlea · Accepted solution]

You could try doing a concatenation of the fields in both the main alert, and also the summary index sub search and then compare those concatenated fields to determine if the results of the current alert were found in the summary index. 

(Also, side note; you may need to enter a hardcoded earliest and latest time in your summary index sub search to ensure that you are looking at the correct time range for the summary indexed data)

You could try something like the following:

index=preos host IN(*) *GPU*: PCISLOT*
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| eval concat_field=host.gpu.VBIOS_Version
| join type=outer concat_field
    [search index=summary   summary_type=test 

| table gpu orig_host VBIOS_Version 
| eval concat_field=orig_host.gpu.VBIOS_Version
| eval is_found_in_summary_index="true"
| table concat_field is_found_in_summary_index]

[vrmandadi]

@jdunlea Thank you for the reply..Yes I am doing summary index because its faster and going forward it becomes easy and fast if I want to increase my search window on summary index to 7 days or more