Splunk stop runing the alert script after a few days, despite that the search trigger works!

[bugnet]

Hi all,
I have deployment environment with:
5 search heads, 3 Indexers, 2 Heavy forwarders and 1 cluster master.

I created the alert on one of the search head with "Cron schedule" and "Run a script" action.
The problem begins after a few days when suddenly the script doesn't run!

What checked:
1. search lookup who trigger the script does work.
2. index=_internal sourcetype="splunk_python" | search "my_script_name" - Does not appear!
Splunk stop running the script after a few days, despite that the search trigger works!
Any insight? What steps do we need to take to the run a script function work in such deployment environment ?

[puneethgowda]

check the expiry option

[gjanders]

I've hit a similar issue to the one you mention in an older Splunk version, re-creating the alert again (with identical configuration) fixed the issue, also upgrading to the current Splunk release (6.5.1) has not resulted in a re-occurence of the issue.

In my case the alert script failed to fire sometimes, even though the alert itself fired, it's not a common problem so it may or may not be the same as the issue you are describing...just a potential answer.

[DalJeanis]

Okay, so if the script worked for a few days, then stopped working, I'd start out by checking whether the components (bin etc) were all in place.

There are lots of production environments where the security team will UNDO work which has not been properly documented, without giving notice. There are also lots of pseudo-production environments where people muck with each other's code, also without giving notice.

[bugnet]

Thanks. all components are in place.