Is there a way to periodically restart Splunk?

[hunterpj]

I need to restart splunk (or reload_ds) every Monday at 7:00AM, as new alerts and dashboards are being made in the files every week. I am tired of manually doing this every week, and was thinking of writing a program to do this for me. But before I do anything, I would like to ask, is there a quicker or easier way to restart splunk on a weekly basis?

[MuS]

Hi hunterpj,

I would strongly recommend against doing an automation of restarting Splunk. I saw customers doing such a thing, and running into troubles because of rouge scripts, or forgot about it.....

If I have to restart Splunk remotely and controlled, I use this approach https://answers.splunk.com/answers/529270/after-deploying-apps-using-the-deployment-server-d.html

cheers, MuS

[hunterpj]

Would doing a reload_ds be fine on a weekly basis? That only refreshes the configuration files if I recall correctly.

[MuS]

A reload deploy-server will update any changes in the Apps/TA's and the deployment client will get the updated Apps/TA's. If the Apps or some App in your serverclasses is configured to restart Splunk it will also restart Splunk after the deployment.

Hope this makes sense ...

cheers, MuS

[niketn]

@hunterpj, is this weekly restart to allow Splunk deployments? Have you considered debug/refresh?

Refer to documentation: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationfilechangesthatrequirerestart...

Also check out Debug Regresh add on built by @MuS

[MuS]

Just an addition here: the debug refresh app should not be use in any production system (as mentioned in the app readme), because it will kill all TCP/UDP inputs regardless which can lead to event loss.

cheers, MuS

[sudosplunk · Accepted solution]

In linux/unix, you can do this by running scripts with cron schedule. Please be sure to modify per your needs.

Below are the two scripts which I use:

This script should be invoked from deployment server.

reload_deploy_server.sh

#!bin/bash

## Variables
date=`date +%Y-%m-%d:%H:%M:%S`
user=`whoami`
hostname=`hostname`
info='INFO'
error='ERROR'
success='SUCCESS'
fail='FAIL'
reload_deploy='reload'
workdir='/opt/splunk/scripts/'
logfile='/opt/splunk/logs/log_for_scripts.log'
userpass='your_admin_password'

## writes event in log file.
echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $user $hostname $reload_deploy msg=\"Initiated reload deploy-server\"" >> $logfile

/opt/splunk/bin/splunk reload deploy-server  -auth admin:$userpass --answer-yes
if [ $? -eq 0 ];
        then
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $success $user $hostname STATUS msg=\"Reloading server classes\"" >> $logfile
        else
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $error $fail $user $hostname STATUS msg=\"Encountered some errors while reloading server classes\"" >> $logfile
fi

This script should be invoked from UFs.

restart_splunk.sh



## Variables
 date=`date +%Y-%m-%d:%H:%M:%S`
 user=`whoami`
 hostname=`hostname`
 info='INFO'
 error='ERROR'
 success='SUCCESS'
 fail='FAIL'
 restart='restart'
 workdir='/opt/splunk/scripts/'
 logfile='/opt/splunk/logs/log_for_scripts.log'

/opt/splunk/bin/splunk restart --answer-yes

/opt/splunk/bin/splunk status
if [ $? -eq 0 ];
        then
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $success $user $hostname STATUS msg=\"Splunk is running\"" >> $logfile
        else
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $error $fail $user $hostname STATUS msg=\"Splunk is not running\"" >> $logfile
fi

Once scripts are in place, configure the crontab as below:

## Deployment server
0 7 * * 1 /opt/splunk/scripts/reload_deploy_server.sh

## Universal forwarders
0 7 * * 1 /opt/splunk/scripts/restart_splunk.sh

[skoelpin]

Whats the purpose of restarting Splunk every week?

[thambisetty]

AFAIK, there is no such option in Splunk. May be you will have to write batch or bash script to do this. Would be better way to do this.

Any how its not big deal. Its just matter of singlei line cronjob in linux.