Knowledge Management

How to create an alert by comparing with previous alert sent to summary index?

vrmandadi
Builder

I am trying to create an alert and send the alert details to summary index.Below is the search I am using.I have scheduled the below search everyday at 2AM and look for yesterday data and send alert and then send same data to summary index..I am trying to create another alert to compare the data with summary index and send alert only if there is a difference in results..I am trying to compare the combination of host gpu and VBIOS_Version fields..if all these are different then send an alert

 

Query for alert

 

index=preo  host IN(*) 
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version |collect index=summary marker="summary_type=test"
| eval details= "preos Splunk: ".host.  " node VBIOS mismatch " .gpu.  " " .VBIOS_Version. " Common:" .Common_VBIOS_Version." date:" .time 
| table details

 

 

Below is the query I tried to compare with summary index and send if there is a change 

 

index=preos host IN(*) *GPU*: PCISLOT*
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| join host gpu VBIOS_Version 
    [search index=summary   summary_type=test 

| table gpu orig_host VBIOS_Version 
| rename orig_host as host ]

 

Labels (1)
0 Karma
1 Solution

jdunlea
Contributor

You could try doing a concatenation of the fields in both the main alert, and also the summary index sub search and then compare those concatenated fields to determine if the results of the current alert were found in the summary index. 

(Also, side note; you may need to enter a hardcoded earliest and latest time in your summary index sub search to ensure that you are looking at the correct time range for the summary indexed data)

 

You could try something like the following:

 

index=preos host IN(*) *GPU*: PCISLOT*
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| eval concat_field=host.gpu.VBIOS_Version
| join type=outer concat_field
    [search index=summary   summary_type=test 

| table gpu orig_host VBIOS_Version 
| eval concat_field=orig_host.gpu.VBIOS_Version
| eval is_found_in_summary_index="true"
| table concat_field is_found_in_summary_index]

 

View solution in original post

johnhuang
Motivator

If you're just comparing yesterday to today's change, you can probably skip using the summary index table.

 

 

index=preos host IN(*) *GPU*: PCISLOT* earliest=-2d@d latest=-0d@d
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| eval event_date=strftime(_time,"%Y-%m-%d")
| stats latest(_time) AS _time latest(*) AS * BY event_date host gpu
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time event_date host gpu ios common_ios gpu_ios 
| eval event_time=strftime(_time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| fillnull value=""
| eval record_hash=MD5(host ." | ".gpu." | ".VBIOS_Version)
| eventstats min(_time) AS first_seen BY record_hash
| where first_seen>relative_time(now(), "-1d@d")
| eval first_seen_time=(first_seen,"%Y-%m-%d %H:%M:%S")
| table event_time gpu host VBIOS_Version

 

 

 

0 Karma

jdunlea
Contributor

That will work too.

 

However if there is a lot of data in the index, then it will mean you have to search double the data by expanding the time range from 1 day to 2 days.  Additionally, it essentially means that you are "double searching" at least one day's worth of data. 

 

On the other hand, there are also technical constraints to using the subsearch and join method.

Just worth noting the tradeoffs between both options. 

johnhuang
Motivator

@jdunlea, yep agreed, the best approach is scenario based.

There's a mistake in the query which write to the summary index. In particular, the _time field was renamed to time and as a result, the timestamp of data saved to the summary index would no longer be accurate.

Hence, not using the summary index data was the most straightforward approach.

 

jdunlea
Contributor

You could try doing a concatenation of the fields in both the main alert, and also the summary index sub search and then compare those concatenated fields to determine if the results of the current alert were found in the summary index. 

(Also, side note; you may need to enter a hardcoded earliest and latest time in your summary index sub search to ensure that you are looking at the correct time range for the summary indexed data)

 

You could try something like the following:

 

index=preos host IN(*) *GPU*: PCISLOT*
| rex field=_raw "log-inventory.sh\[(?<id>[^\]]+)\]\:\s*(?<gpu>[^\:]+)\:\s*(?<Hardware_Details>.*)"
| rex field=_raw "GPU.*PCISLOT.*VBIOS\:\s(?<ios>[^\,]+)"
| search gpu=GPU*
| eval gpu_ios=gpu." : ".ios
| stats latest(_time) AS _time latest(*) AS * BY host gpu
| bucket _time span=1m
| bucket _time span=1m
| appendpipe [| top 1 ios BY _time host | rename ios AS common_ios | table _time common_ios host]
| eventstats max(common_ios) AS common_ios values(gpu_ios) AS gpu_ios BY _time host
| table _time host gpu ios common_ios gpu_ios 
| rename _time as time 
| eval time=strftime(time,"%Y-%m-%d %H:%M:%S")
| rename ios AS VBIOS_Version common_ios as Common_VBIOS_Version gpu_ios as GPU_VBIOS 
| where LEN(gpu)>1 AND VBIOS_Version!=Common_VBIOS_Version 
| eval concat_field=host.gpu.VBIOS_Version
| join type=outer concat_field
    [search index=summary   summary_type=test 

| table gpu orig_host VBIOS_Version 
| eval concat_field=orig_host.gpu.VBIOS_Version
| eval is_found_in_summary_index="true"
| table concat_field is_found_in_summary_index]

 

vrmandadi
Builder

@jdunlea Thank you for the reply..Yes I am doing summary index because its faster and going forward it becomes easy and fast if I want to increase my search window on summary index to 7 days or more

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...