topic Re: coldtofrozen stopped working how to troubleshoot in Monitoring Splunk

How to troubleshoot when coldtofrozen stops working?

mikefg — Mon, 28 Aug 2023 18:25:51 GMT

My coldtofrozen has stopped working. Might be related to python3, but I'm not 100% sure. I've done some tweaking to the coldtofrozen.py #! /opt/splunk/bin python and I've checked other settings, but all seem to be okay.

Are there any commands or tools I can run to help troubleshoot?
Where would the errors be logged?

Thanks

Re: coldtofrozen stopped working how to troubleshoot

isoutamo — Fri, 25 Aug 2023 06:25:08 GMT

it's hard to say what is wrong as we cannot see your script.

Are you update splunk and change python to version 3 or how it have gone broken?

How you are calling it?

Basically you could test it with command under splunk user

splunk cmd <your script>

This runs it like inside splunk as it runs when it normally works.

If this didn't give you any error then just add debug to your script and try again.

r. Ismo

Re: coldtofrozen stopped working how to troubleshoot

mikefg — Fri, 25 Aug 2023 14:29:57 GMT

Thanks for replying.

I had been running splunk 8.2 with it working fine and then upgraded to 9.0.3 on CentOS 7. I'm pretty sure 9.0.3 had been working fine as well, but I may be mistaken on the timing.

I did have to do a server name change on the indexers, but the frozen server stayed the same. My indexers write to a frozen server over NFS and nfs exports uses ip address, not name.

My coldtofrozen.py script hasn't changed, but when I run it this is what I get.

[bin]# ./splunk cmd "/opt/splunk/bin/python" "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py"
usage: python coldToFrozenExample.py <bucket_dir_to_archive>
[bin]#

Full script, edited to remove comments and internal server names.

#!/opt/splunk/bin python

import sys, os, gzip, shutil, subprocess, random, socket
hname = socket.gethostname()

ARCHIVE_DIR = os.path.join('/mnt/nfs/frozensrv', hname)

def handleNewBucket(base, files):
   print('Archiving bucket: ' + base)
   for f in files:
      full = os.path.join(base, f)
      if os.path.isfile(full):
         os.remove(full)

if __name__ == "__main__":
if len(sys.argv) != 2:
sys.exit('usage: python coldToFrozenExample.py <bucket_dir_to_archive>')

   if not os.path.isdir(ARCHIVE_DIR):
         try:
            os.mkdir(ARCHIVE_DIR)
         except OSError:
   # Ignore already exists errors, another concurrent invokation may have already created this dir
   sys.stderr.write("mkdir warning: Directory '" + ARCHIVE_DIR + "' already exists\n")

   bucket = sys.argv[1]
   if not os.path.isdir(bucket):
      sys.exit('Given bucket is not a valid directory: ' + bucket)

   rawdatadir = os.path.join(bucket, 'rawdata')
   if not os.path.isdir(rawdatadir):
         sys.exit('No rawdata directory, given bucket is likely invalid: ' + bucket)

   files = os.listdir(bucket)
   journal = os.path.join(rawdatadir, 'journal.gz')
   if os.path.isfile(journal):
         handleNewBucket(bucket, files)
   else:
         handleOldBucket(bucket, files)

if bucket.endswith('/'):
bucket = bucket[:-1]

indexname = os.path.basename(os.path.dirname(os.path.dirname(bucket)))
destdir = os.path.join(ARCHIVE_DIR, indexname, os.path.basename(bucket))

   while os.path.isdir(destdir):
         print('Warning: This bucket already exists in the archive directory')
         print('Adding a random extension to this directory...')
         destdir += '.' + str(random.randrange(10))

shutil.copytree(bucket, destdir)

Re: coldtofrozen stopped working how to troubleshoot

mikefg — Fri, 25 Aug 2023 20:39:33 GMT

It looks like this line

if len(sys.argv) != 2:

is where the script runs into trouble. I've also tried

if len(sys.argv) < 2:

but same results. Not sure what values I need to pass or where to pass them to get the script to proceed. Maybe something in indexes.conf on the coldToFrozenScript line ?

coldToFrozenScript = "/opt/splunk/bin/python" "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py"

Re: coldtofrozen stopped working how to troubleshoot

isoutamo — Sat, 26 Aug 2023 23:04:35 GMT

When you are running this from cmd line, you must give all needed parameters for it. Check from your script what those are, but if I recall right at least the bucket me it’s path is needed.

Re: coldtofrozen stopped working how to troubleshoot

mikefg — Mon, 28 Aug 2023 16:25:56 GMT

I'm passing a bucket path and getting this error now. Replaced some path values to hide internal names and bucket name.

bash-4.2$ /opt/splunk/bin/splunk cmd "/opt/splunk/bin/python" "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py" "/opt/splunk/var/lib/splunk/indexname/db/db_string"

Traceback (most recent call last):
File "/opt/splunk/etc/peer-apps/archive_app/coldToFrozen.py", line 51, in <module>
handleOldBucket(bucket, files)
NameError: name 'handleOldBucket' is not defined

Re: coldtofrozen stopped working how to troubleshoot

isoutamo — Mon, 28 Aug 2023 16:44:40 GMT

As we don’t know content of your script, we cannot really help you more. I propose that you try to find someone who knows enough Python and you look together what’s wrong in script and fix it.

Re: coldtofrozen stopped working how to troubleshoot

mikefg — Mon, 28 Aug 2023 18:35:21 GMT

I posted my script in this thread and it's the sample script, just edited for my environment. It's been working fine for a long time until this issue arose.

Re: coldtofrozen stopped working how to troubleshoot

mikefg — Mon, 28 Aug 2023 20:56:06 GMT

I got it working again. I had to recopy the coldtofrozenexample.py and edit for my environment. I was missing some sections from the script.