I had this issue today with a real time search (Splunk enterprise v6.2.0), we were experiencing issues where the search terms weren't being picked up by the real time search/alert and alerting on terms that we were excluding. After searching Answers & Docs repeatedly and even Googled the issue, I found nothing.
After much searching/changing search terms, testing & nashing of teeth, I turned on the "List in Triggered Alerts" and then examined the next alert that came up in Job Inspector. I saw that the search terms that I put in the alert were there, but the search job properties did not have the changed search terms, so I was finally on a hot trail. I tried changing the search terms several times, but the changes never made it to the search job properties that are what the search head sends to the indexer.
When I went to the Activity > Jobs menu and went to look at the particular user & Running jobs, I saw a number of zombie processes out there. When I looked at them in Job Inspector, I saw that they had the very search terms that I was trying to change. So, instead of restarting the Splunk instance, I tried finalizing the job / alert that was running (real time). The zombie processes evaporated (and stopped eating my cpu brains!) then the job started back up and it was using the correct, changed search terms. I tried changing the search terms a few times after that and the running job correctly reflected the changes.
I hope this will help someone as I spent about 5 hours messing with this, but it is a good lesson learned. I wasn't aware that zombie processes could prevent changes, although it makes sense. I'll have to use the Job Inspector weapon in more often to rid my installation of zombies.
... View more