You have to somehow pass triggerActions=1 when the search job get's created. Otherwise, as you've seen, the search will run as normal, but without causing any of the triggered actions (like running a script, emailing, or summary indexing) to occur. Normally, this is what you want. For example, you wouldn't want to accidentally click "run" on a summary index generating search that would end up doubling up your summary index data. In the case, of sending an email though, that would be nice to be able to do.
I don't know a way to do this from within the user interface, but I've done it from some python code before, here's a stripped down version that works on my system. Obviously, you'll need to update it to match your parameters. You can change "now" to any epoch time, to re-test at a specific time after tweaking the alert's logic.
import splunk.auth
import splunk.saved
import time
sessionKey = splunk.auth.getSessionKey("admin", "changeme")
now = time.time()
job = splunk.saved.dispatchSavedSearch("Savesearch name", sessionKey=sessionKey, namespace="App", owner="admin", triggerActions=1, now=now)
The REST API docs also show an example using curl:
curl -k -u admin:pass \
https://localhost:8089/servicesNS/admin/search/saved/searches/MySavedSearch/dispatch \
-d trigger_actions=1
... View more