Monitoring Splunk

How to turn off splunkd during certain hours

rholm01
Explorer

I have a customer who wants to have the splunk forwarder turned off during certain critical processing time.

0 Karma
1 Solution

inventsekar
Super Champion

You have to write a cronjob to do this.
To turn off splunk at certain time, you have to write a small script which will include splunk stop and start commands
or two separate cronjobs.

https://tecadmin.net/crontab-in-linux-with-20-examples-of-cron-schedule/

Schedule a cron to execute on every Sunday at 5 PM.
This type of cron are useful for doing weekly tasks, like log rotation etc.

#to start splunk again at 5pm sunday
0 17 * * 7 /opt/splunk/bin/splunk stop

#to start splunk again at 6pm
0 18 * * 7 /opt/splunk/bin/splunk start

View solution in original post

s2_splunk
Splunk Employee
Splunk Employee

Wait, what? They want to have log collection/forwarding turned off during critical processing times? Seems a bit counter-intuitive to me. I would want to know if that critical processing caused any error messages as quickly as possible. What is your customer's concern? Are they worried the forwarder (a universal forwarder, I assume?) will delay their critical processes by 'stealing' too much CPU/memory?
Are their servers running at a CPU utilization well beyond 75% or so during those times?
Did they experience impact caused by the forwarder?
Those are the kinds of questions I would ask, because - ideally - you do not want to stop the forwarder for extended periods of time, especially on a system that creates a lot of log files that potentially roll quickly during higher utilization periods.

If they can't be convinced to not do that kind of thing, cron is your friend.

rholm01
Explorer

ssievert - Loved your response, and your time is much appreciated. I will pass this along to my customer. Thank you!!

0 Karma

inventsekar
Super Champion

You have to write a cronjob to do this.
To turn off splunk at certain time, you have to write a small script which will include splunk stop and start commands
or two separate cronjobs.

https://tecadmin.net/crontab-in-linux-with-20-examples-of-cron-schedule/

Schedule a cron to execute on every Sunday at 5 PM.
This type of cron are useful for doing weekly tasks, like log rotation etc.

#to start splunk again at 5pm sunday
0 17 * * 7 /opt/splunk/bin/splunk stop

#to start splunk again at 6pm
0 18 * * 7 /opt/splunk/bin/splunk start

View solution in original post

sduff_splunk
Splunk Employee
Splunk Employee

Do you want Splunk to stop forwarding during this time, or stop collecting logs during this time? If you stop Splunk, and then start it up at a later time, it will "catchup" on the data that was missing.

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!