99% sure this is a bug. It seems the
stats commands breaks the tokenization of hidden fields. Per the below, and per other users I have talked to on 7.3, this issue exists from at least version 7.0 forwards.
@the_tick mentioned maybe using field aliases to workaround, or
sendalert. I haven't tried either of those workarounds yet, but this was my solution:
| makeresults | eval a="mysubject" | eval b="mycontent" | map maxsearches=10000 search="| makeresults | eval b=\"$b$\" | table b | sendemail server=smtp.gadsden.com to=\"firstname.lastname@example.org\" subject=\"$a$\" message=\"Results:\" sendresults=true inline=true"
To adapt this template to your search, simply calculate the fields that you'll want to tokenize, before the map, and dedup to only be one row of data (e.g. you probably won't be using
makeresults). Next, in the map, write your actual alert and in
sendemail you can now use the calculated tokens from before the map.
This is a sanitized example of my query:
index=wineventlog (EventCode=4771 OR EventCode=4776 AND Keywords="Audit Failure") user=amkfk NOT Source_Workstation="TABLET3849" NOT Source_Workstation="TOKYO_OFFICE" NOT Source_Workstation="MKK_SURFPRO7"| rex field=_raw "Error Code:\s+(?<error_code>\S+)" | stats values(error_code) AS error_codes | eval error_codes=mvjoin(error_codes,",") | table error_codes | map maxsearches=10000 search="search index=wineventlog (EventCode=4771 OR EventCode=4776 AND Keywords=\"Audit Failure\") user=amkfk NOT Source_Workstation=\"TABLET3849\" NOT Source_Workstation=\"TOKYO_OFFICE\" NOT Source_Workstation=\"MKK_SURFPRO7\" | rex field=_raw \"Error Code:\s+(?<error_code>\S+)\" | sort 0 _time | table _time user EventCode Source_Workstation ComputerName Keywords error_code | sendemail server=smtp.mydomain.com to=\"email@example.com\" subject=\"Splunk Alert: amkfk_4771_4776 ($error_codes$)\" message=\"Results:\" sendresults=true inline=true"