Assuming your using Linux, you could 'strace' the splunkd process for a short bit and see what it is doing. strace -p 'PID of splunkd' -o output.txt Let it run for a few minutes then check output.txt. ... View more

I've seen this same question come up a couple times, and my solution is different, so thought I'd share on a few of these in case others have the same problem I did. The problem was that the query in my Alert was "search index=myindex sourcetype=waf httpstatus=400". As soon as I removed the keyword "search" from the beginning of this query in the alert, it produced results consistent with manually issuing the search (index=myindex sourcetype=waf httpstatus=400). The rationale behind this (if I understood the support engineer correctly) is that the Alert passes the query to the CLI (i.e. /bin/splunk search ), so the CLI interprets the "search" item in my query as a searchable word, not a function. ... View more