Very Large Diag File

[reed_kelly]

It is becoming harder to submit cases, because our diag files have gotten very large. In the most recent case, the diag-xxxx-2012-06-12.tar.gz was about 570 MB. A lot of that is Hosts.data files extracted from the db folders. We frequently use the metadata commands for host lists per index, so we don't want to get rid of these as a rule, but having them bloat the diag file is not helpful.

I can unpack the tar.gz file and remove the Hosts.data files, but I was wondering how others have dealt with large diag files. Also, the files are still pretty large after removing Hosts.data.

[pkumar9610]

I have similar issue, I am seeing Diag tgz file as 14GB where it was 3GB couple of months back. Not sure what is causing this issue, I have also tried excluding *.data files but still it didn't help.

Any inputs ?

[marty_lindsay]

need the correct spelling of exclude:

./splunk diag --exclude *.data

[trumpdeck · Accepted solution]

Here are some techniques to reduce the size of the diag :

If you do, please always mention in the case that files are missing from the diag.

• Check if they are no dump or core files in $SPLUNK_HOME/var/log/splunk/*.dmp If this is the case, move them out of the folder before generating the diag, and upload separately the last one only.
• If on the indexers the biggest file are your metadata (hosts.meta, sources.meta, sourcetype.meta etc...), then you may have a issue with performance (the SOS app warning dashboard will tell you).
• You can exclude them from the diag, and upload them separately example splunk diag --exlude *.data see http://docs.splunk.com/Documentation/Splunk/4.3.2/Troubleshooting/ContactSplunkSupport

[tpsplunk]

another thing to watch out for is if your splunk server uses disk storage served up by a SAN/NAS that is using storage snapshots. The diag process may try to include them. in my case we use NetApp and splunk diag was picking up a bunch of files in .snapshot which bloated my diag file to 3GB. Support and I tracked this down by examining the contents of the 3GB tar file like so: tar ztvf diagfile.tar.gz | sort -k3 -r -n > /tmp/sorted-tar-contents.txt && less /tmp/sorted-tar-contents.txt. once we knew we were there we could exclude them using the aforementioned diag --exclude option