Why is the Splunk ExecProcessor Rescheduling a specific cronjob every interval?

[mitchfultz]

Hello,

Splunk ExecProcessor is rescheduling a specific cronjob for the Rapid7 Nexpose add-on; rapid7nexpose.py is being rescheduled at nearly every interval, even if I set it to run every minute. From splunkd.log:

ExecProcessor - setting reschedule_ms=3599999, for command=python /opt/splunk/etc/apps/TA-rapid7_nexpose/bin/rapid7nexpose.py

I have checked and all files are splunk:splunk with the following permissions: -rwxr-xr-x
Authentication with the Nexpose device is successful since we do occassionally get a job to run and fetch the logs. Any help would be greatly appreciated as Splunk support will not assist.

[shartwell]

Hi mitchfultz -

I've looked into this extensively as I'm having the same issue - it appears the rapid7nexpose.py process doesn't exit at the end of its run against the sites eligible for query. The process below started at 5:00am and is still running despite reaching the end of the list of sites to process:

$ ps -ef | grep -v grep | grep nexpose
splunk 58152 29727 0 05:00 ? 00:00:18 python /opt/splunk/etc/apps/TA-rapid7_nexpose/bin/rapid7nexpose.py

Sadly, I've resorted to issuing a kill to take it down after several hours. The splunk daemon restarts it automatically and it begins another run without any further intervention. I've setup a cron job to run once an hour and kill this process if its older than 4 .5 hours - you may need to adjust the time based on number of sites and typical duration:

$ crontab -l
0 */1 * * * kill $(find /proc -maxdepth 1 -user splunk -type d -mmin +270 -exec basename {} \; | xargs ps | grep nexpose | awk '{ print $1 }')

Not a permanent fix, but it will keep me from restarting it manually constantly until I find out why it continues to be rescheduled into the future.