A scheduled job runs 2 times due to replication between Search Heads. User gets 2 emails. Want to run only once

Path Finder


A user scheduled job to run & email him results. When fired off, He gets 2 emails on each run. This is not good.

I did locate his job scheduled on BOTH our Search Head-1 & Search Head-2 svrs via Settings > S&R.

Background: When we installed splunk- Our consultant setup replication between Search Heads for redundancy. This is good, except that the job gets mirrored onto SH2 also, thus creating a 2nd run, thus problem.

How do I manage jobs so that they run only once, while still maintaining the redundancy required. This must be a common problem of lots of duplication.

Thank you in advance for ideas to solve.


Tags (1)
0 Karma


You can disable the scheduler on one of the search heads:

0 Karma

Path Finder

Thx for the quick response Wolverine - Grazie!

So, I put in your suggestion:
index=_internal source=*scheduler.log
in both SH-1 & SH-2 - but it just spins on both SH's...runs forever & no hits at all. So, that can't be good/right.

I also tried:
index=_internal host=mdc2brc0123 source="/opt/splunk/var/log/splunk/scheduler.log"

spins also.

So, why would it show Enable if its really disabled? Why can't it report what is tru? 😉 I've been knocking myself out here running in circles b/c I can't count on what i'm seeing. Frustrating for sure.

I'm stuck big time.

0 Karma


You'll see jobs enabled but they won't run on both SHs. To verify check the index=_internal source=*scheduler.log. You should only see searches running from one SH, not both.

Also, you should be able to see all saved searches and reports from either SH since they are pooled right? If you do not have SH_POOL set up then you need to do this so you do not have redundant SHs.

0 Karma

Path Finder

I tried your suggestion - but I cannot tell if it worked or not.Here is what I did:
Note: SearchHead-1 & SearchHead-2 both run a job scheduler. I want to turn off job scheduler completely on SearchHead-2. Steps:
ssh SH-2
vi /opt/splunk/etc/system/default/default-mode.conf
disabled = true

/opt/splunk/bin/splunk restart (no issues)

Now, how do I gaurentee JS is Really not running on SH-2? I checked SH-2 Settings>S&R>any/any and I still see all jobs listed as ENABLED. Need help, I am getting slammed by customers complaining about still receiving 2x emails per job.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!