Monitoring Splunk

Change output location of splunk diag?

smisplunk
Path Finder

I've seen from personal experience, as well as the answer at http://answers.splunk.com/questions/167/how-do-i-generate-a-diag-splunk-diagnostics-file that the output from a splunk diag command goes to $SPLUNK_HOME. One of our users requests that it be placed in /var. Is this possible?

Tags (1)
1 Solution

Lowell
Super Champion

It does not appear that specifying an output location is a supported feature in the current release. If your issues is disk space, you should also be aware that a temporary copy of all the files that are being archived are also kept under your $SPLUNK_HOME location as well. (The temporary folder is $SPLUNK_HOME/var/run/splunk/diag-temp)

Your can review the splunk diag actions in the following script: $SPLUNK_HOME/lib/python2.6/site-packages/splunk/clilib/info_gather.py

Note: This is the unix install path, it may be slightly different on Windows or older version of splunk

I took a quick look. If you really want, you could override the get_tar_name() function to return a different path, but making modifications definitely falls into the "at-your-own-risk" category. Each upgrade would overwrite any customizations.

If this is something that you need long term, I would suggest sending a feature request to splunk support. It doesn't seem like it should be all that difficult to implement.

View solution in original post

JL99
Explorer

I use a workaround. Background of what I did: I am writing a bash script, to create Splunk diag using splunk user and move the diag file to the desired folder of mine.

[root@myserver ~]# vi test_script.sh
sudo -i -u splunk bash << EOF
# Create Splunk Diagnostic Report
/opt/splunk/bin/splunk diag > /opt/splunk/mylog.log
EOF

# Use grep to extract the path from the output
diag_path=$(cat /opt/splunk/mylog.log | grep -oP '(?<=Splunk diagnosis file created: )/.*\.tar\.gz')
echo $diag_path

# Check if the path is not empty
if [ -n "$diag_path" ]; then
     # Copy the file to /root/mydesiredfolder
     mv "$diag_path" /root/mydesiredfolder
     echo "File copied successfully to /root/mydesiredfolder"
else
     echo "Path not found or command did not generate the expected output"
fi

# Cleanup
rm /opt/splunk/mylog.log

##########
The idea behind it is when running ./splunk diag, it will have the output of something like this
Splunk diagnosis file created: /opt/splunk/diag-servername-2023-12-22_08-19-01.tar.gz

0 Karma

lumpymilk
Explorer

It's probably worth mentioning since it was difficult to find. There is an option to write the diag to stdout as an uncompressed tar. From there you can do what you like. This is useful if you don't have space on the remote server.

this will send the diag as uncmpressed tar to stdout, then pass that through gzip prior to passing the gzipped output back through your ssh session and redirecting to a local gzipped copy. Note, the pipe needs to be escaped if you want to gzip prior to data transferring across the network.
ssh user@remoteserver $SPLUNK_HOME/bin/splunk diag --stdout | gzip > mydiag.tgz

If you just want to put the file into /tmp or /var then you can do this...
$SPLUNK_HOME/splunk diag --stdout | gzip > /tmp/mydiag.tgz

0 Karma

wbfoxii
Communicator

Splunk 6 allows you to grab a diag file remotely. This saved me today as I was writing a diag with debug information that was too large for the remaining space in /opt/splunk/ on the server.

bin/splunk/diag -uri https://splunk-server:mgmt-port

brings the diag file to your current server.

0 Karma

Lowell
Super Champion

It does not appear that specifying an output location is a supported feature in the current release. If your issues is disk space, you should also be aware that a temporary copy of all the files that are being archived are also kept under your $SPLUNK_HOME location as well. (The temporary folder is $SPLUNK_HOME/var/run/splunk/diag-temp)

Your can review the splunk diag actions in the following script: $SPLUNK_HOME/lib/python2.6/site-packages/splunk/clilib/info_gather.py

Note: This is the unix install path, it may be slightly different on Windows or older version of splunk

I took a quick look. If you really want, you could override the get_tar_name() function to return a different path, but making modifications definitely falls into the "at-your-own-risk" category. Each upgrade would overwrite any customizations.

If this is something that you need long term, I would suggest sending a feature request to splunk support. It doesn't seem like it should be all that difficult to implement.

smisplunk
Path Finder

Good idea. I'll file a feature request.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...