Getting Data In

Restart a UF via CLI / other remote means

Communicator

Hi

Is there a way to remotely restart a UF forwarder in splunk directly from within splunk e.g. using splunk cli or some kind of splunk REST API command or splunkweb funtionality.

I couldn't find anything about this and was curious to know if this was possible at all 🙂

0 Karma

SplunkTrust
SplunkTrust

The TL;DR version of this is in Splunk 6.5.2 on AIX (and we also had what appeared to be the same issue on Linux):

/opt/splunkforwarder/bin/splunk restart

Called from a script run by the splunkforwarder (eg. a bin/script.sh that is within an app deployed to the splunk forwarder) left the forwarder in a shutdown state.

The following also failed, even after getting the process to run post-shutdown of the forwarder:

/opt/splunkforwarder/bin/splunk restart
/opt/splunkforwarder/bin/splunk restart > /tmp/test.txt
/opt/splunkforwarder/bin/splunk restart 2>&1

The only version that worked was:

/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1

I am unsure why, but the above works consistently, it does not matter if the output goes to /dev/null or to a file, more details below.

/opt/splunkforwarder/bin/splunk restart command worked fine from the CLI (if the splunk user was logged in), however if we had a splunk application executing a script (in our case one that updated the inputs.conf) and the mentioned script attempted to restart Splunk it would result in the Splunk forwarder stopping and not starting again...
Manual runs of the script while logged in were not an issue so it was something related to this...

After quite a bit of investigation & testing we found that Splunk would not start if the standard and error output of the process was not sent somewhere.
Eventually we found appending a > /tmp/file.txt and a 2>&1 to the restart command resolved the issue, sending the data to /dev/null appears to work the same.

In other words:

/opt/splunkforwarder/bin/splunk restart > /tmp/test.txt 2>&1

I found the above works, however if I do not have > /tmp/test.txt (or /dev/null) or I do not have the 2>&1 the forwarder only gets to this point:

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
        Checking mgmt port [127.0.0.1:8089]: open
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunkforwarder/splunkforwarder-6.5.2-67571ef4b87d-AIX-powerpc-manifest'

And then mysteriously the process stops running, I have this issue on 6.5.2 so hopefully it helps someone else who has needs to restart the forwarder & does not want to put a username/password into their scripts to use the REST API can use this trick...

Alternative suggestions are also welcome, and other answers in this same thread may suit your situation (such as the REST API).
I'm just posting in case this can help someone else, thanks

0 Karma

Explorer

I know this topic may be old but I wanted to add, that this solved my scripted input problem too. 

 

I had a script in which I change some things and then want to restart splunk - only with the statement

/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1

 it works perfectly. otherwise, when I leave 2>&1 out it results in splunk stopping but simultaneously killing the skript process too. so splunk can't come up anymore. so, thanks for that valuable information

0 Karma

Motivator

You might be able to make use of this app: https://splunkbase.splunk.com/app/2775/

great app.

0 Karma

Communicator

If REST API isnt an option, on Windows you can manage the service using native Windows protocols/tools. The service is called something like splunk universal forwarder.

0 Karma

Splunk Employee
Splunk Employee

You can use Web Terminal for Splunk app to execute remote REST API call. Look for example in the documentation tab of the app page.

0 Karma

Contributor

Yes you can do it using the REST API :

https://splunkinstance:8089/services/server/control/restart

Motivator

It gives the below error :

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<response>
<messages>
<msg type="ERROR">
In handler 'server-control': Invalid request, restart requires POST (handler: server-control, action:restart, eai action: list).
</msg>
</messages>
0 Karma

Path Finder

Per the error message, try using POST instead of GET. For example:

curl -X POST -k -u admin "https://localhost:8089/services/server/control/restart"

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!