Repeated calls to Splunk list command on deployment server causes Splunk to crash

[francis_larkin]

I am writing a python based script to monitor splunk via the command line interface. Using cron we call a bash script that call the CLI from bash and write the output to flat files. The python script then is called to read these files, munge them, and create a status file that I make decisions with.

Originally we were calling the cli command using python subprocess but decided to move that outside python and do it in bash. This helped a little but eventually splunk crashed taking the analytics with it.

We are using splunk supplied by our vendor. Our current version is Splunk 4.3.4 (build 136012)

Any help is greatly appreciated....

[francis_larkin]

Problem solved...
I am working with a vendor that provides Splunk as part of their package. This is both good and bad. In the script I developed to monotier and report, I was writing one of my files to a folder that the vendor's software monitored. Whenever I updated that file (once a minute) then vendor's software kicked off a re-deployment on the deployment server. Since I was doing this every minute it eventually caused splunnd to throw up on my job scheduler and anaytics. Not writing to the folder fixed this issue. I am now using the REST interface to gather deploy-clinent, and search-server information one a minute without error.