Hi
Is there a way to remotely restart a UF forwarder in splunk directly from within splunk e.g. using splunk cli or some kind of splunk REST API command or splunkweb funtionality.
I couldn't find anything about this and was curious to know if this was possible at all 🙂
The TL;DR version of this is in Splunk 6.5.2 on AIX (and we also had what appeared to be the same issue on Linux):
/opt/splunkforwarder/bin/splunk restart
Called from a script run by the splunkforwarder (eg. a bin/script.sh that is within an app deployed to the splunk forwarder) left the forwarder in a shutdown state.
The following also failed, even after getting the process to run post-shutdown of the forwarder:
/opt/splunkforwarder/bin/splunk restart
/opt/splunkforwarder/bin/splunk restart > /tmp/test.txt
/opt/splunkforwarder/bin/splunk restart 2>&1
The only version that worked was:
/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1
I am unsure why, but the above works consistently, it does not matter if the output goes to /dev/null or to a file, more details below.
/opt/splunkforwarder/bin/splunk restart command worked fine from the CLI (if the splunk user was logged in), however if we had a splunk application executing a script (in our case one that updated the inputs.conf) and the mentioned script attempted to restart Splunk it would result in the Splunk forwarder stopping and not starting again...
Manual runs of the script while logged in were not an issue so it was something related to this...
After quite a bit of investigation & testing we found that Splunk would not start if the standard and error output of the process was not sent somewhere.
Eventually we found appending a > /tmp/file.txt and a 2>&1 to the restart command resolved the issue, sending the data to /dev/null appears to work the same.
In other words:
/opt/splunkforwarder/bin/splunk restart > /tmp/test.txt 2>&1
I found the above works, however if I do not have > /tmp/test.txt (or /dev/null) or I do not have the 2>&1 the forwarder only gets to this point:
Splunk> The Notorious B.I.G. D.A.T.A.
Checking prerequisites...
Checking mgmt port [127.0.0.1:8089]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunkforwarder/splunkforwarder-6.5.2-67571ef4b87d-AIX-powerpc-manifest'
And then mysteriously the process stops running, I have this issue on 6.5.2 so hopefully it helps someone else who has needs to restart the forwarder & does not want to put a username/password into their scripts to use the REST API can use this trick...
Alternative suggestions are also welcome, and other answers in this same thread may suit your situation (such as the REST API).
I'm just posting in case this can help someone else, thanks
I know this topic may be old but I wanted to add, that this solved my scripted input problem too.
I had a script in which I change some things and then want to restart splunk - only with the statement
/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1
it works perfectly. otherwise, when I leave 2>&1 out it results in splunk stopping but simultaneously killing the skript process too. so splunk can't come up anymore. so, thanks for that valuable information
You might be able to make use of this app: https://splunkbase.splunk.com/app/2775/
great app.
If REST API isnt an option, on Windows you can manage the service using native Windows protocols/tools. The service is called something like splunk universal forwarder.
You can use Web Terminal for Splunk app to execute remote REST API call. Look for example in the documentation tab of the app page.
Yes you can do it using the REST API :
https://splunkinstance:8089/services/server/control/restart
It gives the below error :
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<response>
<messages>
<msg type="ERROR">
In handler 'server-control': Invalid request, restart requires POST (handler: server-control, action:restart, eai action: list).
</msg>
</messages>
Per the error message, try using POST instead of GET. For example:
curl -X POST -k -u admin "https://localhost:8089/services/server/control/restart"