How can I get Splunk_TA_nix to stop running lsof.sh?

[esalesapns2]

I can't figure out why lsof.sh is running every minute.
Here's the "btool inputs list --debug" output for lsof:

/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = 1
/opt/splunkforwarder/etc/system/local/inputs.conf host = c20sbap01l01
/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf index = cre_linux
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf interval = 600
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf source = lsof
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf sourcetype = lsof

Here's my splund.log output:

10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh
10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - interval: 60000 ms

I've tried restarting splunk to no effect...
Notice that the interval is set to 600 (600 seconds) in the btool output, but 60000 (60 seconds) in the splunkd.log output.
I'll try interval = -1 next, and then a single app after that.

[esalesapns2 · Accepted solution]

I had to remove/re-install the splunk_TA_nix app to get it to stop behaving this way.

[corybits]

Thank you for the insights shared, my question is; when you set the indexes for all the Splunk_TA_nix inputs in Splunk_TA_nix/local/inputs.conf, did you use the same index for all the inputs, and was the index/indexes defined in your indexes.conf? Bare with me, new to Splunk.

[woodcock]

The btool is your friend here. You could have an inputs.conf in any app that is causing things so try this:

$SPLUNK_HOME/bin/splunk btool list inputs --debug | grep lsof

[esalesapns2]

Ok, I stopped Splunk, removed the Splunk_TA_nix app, started splunk put the app back and started splunk, and I'm finally no longer getting lsof events. However, I now need to do the same on all my deployment clients... Good thing I was planning on working late.

[esalesapns2]

I should say I restarted Splunk after I put the app back.

[jacobpevans]

Dumb question, but did you run btool on the same machine that the splunkd log is from?

[esalesapns2]

Yes, I did. I have since stopped Splunk, removed deploymentclient.conf, and the DS2-Splunk_TA_nix-cre directory, and set the indexes for all the Splunk_TA_nix inputs in Splunk_TA_nix/local/inputs.conf, and restarted splunk. I'm still getting a steady stream of lsof events every minute.

[esalesapns2]

This is on Splunk Universal Forwarder 7.0.1.

[richgalloway]

Do you want to stop lsof.sh from running at all or just make it run every 10 minutes?

[esalesapns2]

stop it altogether. some of my servers have 5M files open at a time...