Deployment Architecture

Is there a way to periodically restart Splunk?

hunterpj
Path Finder

I need to restart splunk (or reload_ds) every Monday at 7:00AM, as new alerts and dashboards are being made in the files every week. I am tired of manually doing this every week, and was thinking of writing a program to do this for me. But before I do anything, I would like to ask, is there a quicker or easier way to restart splunk on a weekly basis?

0 Karma
1 Solution

sudosplunk
Motivator

In linux/unix, you can do this by running scripts with cron schedule. Please be sure to modify per your needs.

Below are the two scripts which I use:

This script should be invoked from deployment server.

reload_deploy_server.sh

#!bin/bash

## Variables
date=`date +%Y-%m-%d:%H:%M:%S`
user=`whoami`
hostname=`hostname`
info='INFO'
error='ERROR'
success='SUCCESS'
fail='FAIL'
reload_deploy='reload'
workdir='/opt/splunk/scripts/'
logfile='/opt/splunk/logs/log_for_scripts.log'
userpass='your_admin_password'

## writes event in log file.
echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $user $hostname $reload_deploy msg=\"Initiated reload deploy-server\"" >> $logfile

/opt/splunk/bin/splunk reload deploy-server  -auth admin:$userpass --answer-yes
if [ $? -eq 0 ];
        then
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $success $user $hostname STATUS msg=\"Reloading server classes\"" >> $logfile
        else
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $error $fail $user $hostname STATUS msg=\"Encountered some errors while reloading server classes\"" >> $logfile
fi

This script should be invoked from UFs.

restart_splunk.sh



## Variables
 date=`date +%Y-%m-%d:%H:%M:%S`
 user=`whoami`
 hostname=`hostname`
 info='INFO'
 error='ERROR'
 success='SUCCESS'
 fail='FAIL'
 restart='restart'
 workdir='/opt/splunk/scripts/'
 logfile='/opt/splunk/logs/log_for_scripts.log'

/opt/splunk/bin/splunk restart --answer-yes

/opt/splunk/bin/splunk status
if [ $? -eq 0 ];
        then
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $success $user $hostname STATUS msg=\"Splunk is running\"" >> $logfile
        else
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $error $fail $user $hostname STATUS msg=\"Splunk is not running\"" >> $logfile
fi

Once scripts are in place, configure the crontab as below:

## Deployment server
0 7 * * 1 /opt/splunk/scripts/reload_deploy_server.sh

## Universal forwarders
0 7 * * 1 /opt/splunk/scripts/restart_splunk.sh

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi hunterpj,

I would strongly recommend against doing an automation of restarting Splunk. I saw customers doing such a thing, and running into troubles because of rouge scripts, or forgot about it.....

If I have to restart Splunk remotely and controlled, I use this approach https://answers.splunk.com/answers/529270/after-deploying-apps-using-the-deployment-server-d.html

cheers, MuS

hunterpj
Path Finder

Would doing a reload_ds be fine on a weekly basis? That only refreshes the configuration files if I recall correctly.

0 Karma

MuS
SplunkTrust
SplunkTrust

A reload deploy-server will update any changes in the Apps/TA's and the deployment client will get the updated Apps/TA's. If the Apps or some App in your serverclasses is configured to restart Splunk it will also restart Splunk after the deployment.

Hope this makes sense ...

cheers, MuS

0 Karma

niketnilay
Legend

@hunterpj, is this weekly restart to allow Splunk deployments? Have you considered debug/refresh?

Refer to documentation: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationfilechangesthatrequirerestart...

Also check out Debug Regresh add on built by @MuS

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

MuS
SplunkTrust
SplunkTrust

Just an addition here: the debug refresh app should not be use in any production system (as mentioned in the app readme), because it will kill all TCP/UDP inputs regardless which can lead to event loss.

cheers, MuS

sudosplunk
Motivator

In linux/unix, you can do this by running scripts with cron schedule. Please be sure to modify per your needs.

Below are the two scripts which I use:

This script should be invoked from deployment server.

reload_deploy_server.sh

#!bin/bash

## Variables
date=`date +%Y-%m-%d:%H:%M:%S`
user=`whoami`
hostname=`hostname`
info='INFO'
error='ERROR'
success='SUCCESS'
fail='FAIL'
reload_deploy='reload'
workdir='/opt/splunk/scripts/'
logfile='/opt/splunk/logs/log_for_scripts.log'
userpass='your_admin_password'

## writes event in log file.
echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $user $hostname $reload_deploy msg=\"Initiated reload deploy-server\"" >> $logfile

/opt/splunk/bin/splunk reload deploy-server  -auth admin:$userpass --answer-yes
if [ $? -eq 0 ];
        then
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $success $user $hostname STATUS msg=\"Reloading server classes\"" >> $logfile
        else
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $error $fail $user $hostname STATUS msg=\"Encountered some errors while reloading server classes\"" >> $logfile
fi

This script should be invoked from UFs.

restart_splunk.sh



## Variables
 date=`date +%Y-%m-%d:%H:%M:%S`
 user=`whoami`
 hostname=`hostname`
 info='INFO'
 error='ERROR'
 success='SUCCESS'
 fail='FAIL'
 restart='restart'
 workdir='/opt/splunk/scripts/'
 logfile='/opt/splunk/logs/log_for_scripts.log'

/opt/splunk/bin/splunk restart --answer-yes

/opt/splunk/bin/splunk status
if [ $? -eq 0 ];
        then
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $info $success $user $hostname STATUS msg=\"Splunk is running\"" >> $logfile
        else
        echo -e "$(date +%Y-%m-%d:%H:%M:%S) $error $fail $user $hostname STATUS msg=\"Splunk is not running\"" >> $logfile
fi

Once scripts are in place, configure the crontab as below:

## Deployment server
0 7 * * 1 /opt/splunk/scripts/reload_deploy_server.sh

## Universal forwarders
0 7 * * 1 /opt/splunk/scripts/restart_splunk.sh

skoelpin
SplunkTrust
SplunkTrust

Whats the purpose of restarting Splunk every week?

0 Karma

thambisetty
Super Champion

AFAIK, there is no such option in Splunk. May be you will have to write batch or bash script to do this. Would be better way to do this.

Any how its not big deal. Its just matter of singlei line cronjob in linux.

————————————
If this helps, give a like below.
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!