I've been running into an issue where a custom correlation search alert is not returning substitution variables in the Incident Review dashboard.
This custom correlation search alert used to properly return substitution variables $variableXYZ$
Name_of_alert_goes_here Triggered by $srcip$**
**where $srcip$ should be returning an ip address, however, it has been returning "unknown"
All $variable$ calls from this correlation search alert are returning "unknown" when it used to return the variables fine.
**the alert has multiple variable references in the description, drill down link and drill down search
Has anyone experience this issue recently?
Also having this issue. Any variable substitution is replaced with unknown when looking at the notable event. I've verified that the field ($user$,$host$, etc.) does exist in the data within the context of ES.
The docs barely mention this feature, and variable substitution does work for the pre-built correlation searches but I'm not sure how.
Anyone have any insight or ideas?
Working with support, I noticed that ES is not extracting the events properly in index=notable which leads to these unknown values. From what I am seeing, Splunk's extraction needs to be updated because they are expecting field="value" which means value needs to be surrounded by quotes. However, if your value has a backslash character before one of the quotes, the extraction breaks.
Do a search in index=notable and find one of the events that isn't being extracted and look to see if you have anything that would break the extraction.
Could you post the correlation search? Depending on the search, srcip may not be in the results.
My other thought is to include | `map_notable_fields` at the end of your correlation search.
Below is the search query (it's a simple search)
The peculiar thing is, I have another alert based on the below query with difference is it fires when multiple occurrences in a 60 minute period (count > 10) and it works (with variable calls in the incident review dashboard showing the srcip properly)
sourcetype=fortinet type=utm subtype=ips "severity=high" OR "severity=critical" | eval attack=coalesce(attack,attackname) | lookup clientlocation srcip AS srcip OUTPUT location AS src_location contact AS src_contact | lookup clientlocation srcip AS dstip OUTPUT location AS dst_location contact AS dst_contact | eval src_location=if(src_location="OK", "External", src_location) | eval src_contact=if(src_contact="OK", "None", src_contact) | eval dst_location=if(dst_location="OK", "External", dst_location) | eval dst_contact=if(dst_contact="OK", "None", dst_contact) | fillnull value="-" | table _time attack attackid vendor_action severity sessionid user srcip src_location src_contact srcport dstip dst_location dst_contact dstport service hostname sentbyte rcdbyte duration type subtype level policyid _raw | addinfo