Alerting

Ghost alert

the_wolverine
Champion

Has anyone else seen an alert go out when it was not scheduled? I manually scheduled an alert to go out for testing, say 11:15am. Once verified, I scheduled that alert to go out at a specific time (8am, daily.) However, now I'm getting 2 alerts, one for my scheduled time (8am) and again at 11:15am.

In particular this was a scheduled PDF report. I have not seen it happen with a non-PDF Report scheduled alert.

The search was created in UI and scheduled from UI. I scheduled a cron to run at 11:15am to verify that it worked. Then I changed the schedule to ( 0 8 * * * ). It has been running for several days now but I'm getting 2 alerts daily: 1 at 8:15am and another at 11:15am.

I've scoured the filesystem and can find no evidence of where the ghost alert is coming from. The search exists in savedsearches.conf with the ( 0 8 * * * ) schedule.

Tags (2)
0 Karma
1 Solution

the_wolverine
Champion

The issue was resolved by restarting the SHs (we're using a pool). This appears to be some sort of bug (version 4.3.4).

View solution in original post

0 Karma

the_wolverine
Champion

The issue was resolved by restarting the SHs (we're using a pool). This appears to be some sort of bug (version 4.3.4).

0 Karma

Rob
Splunk Employee
Splunk Employee

How did you manually schedule the search? From the manager GUI or from the search itself?

It might be that one search is per user and the other is per app or system.

If you are using a linux box, then you can probably find it in the $SPLUNK_HOME/etc/ directory using something like:

# find . -name savedsearches.conf |grep -i <savedSearchName>
0 Karma
Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...