How to turn off splunkd during certain hours

[rholm01]

I have a customer who wants to have the splunk forwarder turned off during certain critical processing time.

[s2_splunk]

Wait, what? They want to have log collection/forwarding turned off during critical processing times? Seems a bit counter-intuitive to me. I would want to know if that critical processing caused any error messages as quickly as possible. What is your customer's concern? Are they worried the forwarder (a universal forwarder, I assume?) will delay their critical processes by 'stealing' too much CPU/memory?
Are their servers running at a CPU utilization well beyond 75% or so during those times?
Did they experience impact caused by the forwarder?
Those are the kinds of questions I would ask, because - ideally - you do not want to stop the forwarder for extended periods of time, especially on a system that creates a lot of log files that potentially roll quickly during higher utilization periods.

If they can't be convinced to not do that kind of thing, cron is your friend.

[rholm01]

ssievert - Loved your response, and your time is much appreciated. I will pass this along to my customer. Thank you!!

[inventsekar · Accepted solution]

You have to write a cronjob to do this.
To turn off splunk at certain time, you have to write a small script which will include splunk stop and start commands
or two separate cronjobs.

https://tecadmin.net/crontab-in-linux-with-20-examples-of-cron-schedule/

Schedule a cron to execute on every Sunday at 5 PM.
This type of cron are useful for doing weekly tasks, like log rotation etc.

#to start splunk again at 5pm sunday
0 17 * * 7 /opt/splunk/bin/splunk stop

#to start splunk again at 6pm
0 18 * * 7 /opt/splunk/bin/splunk start

[sduff_splunk]

Do you want Splunk to stop forwarding during this time, or stop collecting logs during this time? If you stop Splunk, and then start it up at a later time, it will "catchup" on the data that was missing.