Starting Splunk from post_backup script?

[gljiva]

Hi, I'm having a problem with ARCserve post backup script that should start Splunk again. Prebackup script for stopping Splunk works fine but uag_post_backup can't seem to start Splunk again. I should note that when I manually run script from shell it works fine, but when ARCserve starts script then Splunk has problems starting. Here is uag_post_backup source:

#!/bin/bash

export SPLUNK_HOME=/opt/splunk
date >> /tmp/uag_post.txt
echo "Starting process..." >> /tmp/uag_post.txt

RET=1
while [ $RET = 1 ]
do
    echo "In loop" >> /tmp/uag_post.txt
    id >> /tmp/uag_post.txt
    /opt/splunk/bin/splunk start >> /tmp/uag_post.txt | 2>> /tmp/uag_post.txt
    echo "After start" >> /tmp/uag_post.txt 
    sleep 10
    /opt/splunk/bin/splunk status >> /tmp/uag_post.txt | 2>> /tmp/uag_post.txt
    /opt/splunk/bin/splunk status | grep "splunkd is running"
    RET=$?
    echo "Loop end" >> /tmp/uag_post.txt
    sleep 5
done

This script generates following log file:

Thu Feb 10 11:36:31 CET 2011
Starting process...
In loop
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

Splunk> Take the sh out of IT.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]:  Checking configuration...  Done.
open
After start
splunkd is not running.
splunkweb is not running.
Loop end
In loop

Splunk> Take the sh out of IT.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]:  Checking configuration...  Done.
open
After start
splunkd is not running.
splunkweb is not running.
Loop end
In loop
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

Splunk> Take the sh out of IT.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]:  Checking configuration...  Done.
open
After start
splunkd is not running.
splunkweb is not running.
Loop end

I have removed the rest because this just loops and generates same log because Splunk can't start. Any ideas or ways to start Splunk reliably?

[gljiva · Accepted solution]

The only way to start Splunk from a script without it crashing is to schedule it's start from a script with at command.