topic Re: How can I get Splunk_TA_nix to stop running lsof.sh? in All Apps and Add-ons

How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Sat, 06 Jun 2020 17:53:52 GMT

I can't figure out why lsof.sh is running every minute.
Here's the "btool inputs list --debug" output for lsof:

/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = 1
/opt/splunkforwarder/etc/system/local/inputs.conf host = c20sbap01l01
/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf index = cre_linux
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf interval = 600
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf source = lsof
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf sourcetype = lsof

Here's my splund.log output:

10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh
10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - interval: 60000 ms

I've tried restarting splunk to no effect...
Notice that the interval is set to 600 (600 seconds) in the btool output, but 60000 (60 seconds) in the splunkd.log output.
I'll try interval = -1 next, and then a single app after that.

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

richgalloway — Thu, 10 Oct 2019 19:39:44 GMT

Do you want to stop lsof.sh from running at all or just make it run every 10 minutes?

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Thu, 10 Oct 2019 19:41:52 GMT

stop it altogether. some of my servers have 5M files open at a time...

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

jacobpevans — Thu, 10 Oct 2019 19:55:21 GMT

Dumb question, but did you run btool on the same machine that the splunkd log is from?

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Wed, 30 Sep 2020 02:26:12 GMT

Yes, I did. I have since stopped Splunk, removed deploymentclient.conf, and the DS2-Splunk_TA_nix-cre directory, and set the indexes for all the Splunk_TA_nix inputs in Splunk_TA_nix/local/inputs.conf, and restarted splunk. I'm still getting a steady stream of lsof events every minute.

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Thu, 10 Oct 2019 20:07:54 GMT

This is on Splunk Universal Forwarder 7.0.1.

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Wed, 30 Sep 2020 02:26:19 GMT

Ok, I stopped Splunk, removed the Splunk_TA_nix app, started splunk put the app back and started splunk, and I'm finally no longer getting lsof events. However, I now need to do the same on all my deployment clients... Good thing I was planning on working late.

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Thu, 10 Oct 2019 20:58:03 GMT

I should say I restarted Splunk after I put the app back.

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

woodcock — Mon, 14 Oct 2019 01:22:56 GMT

The btool is your friend here. You could have an inputs.conf in any app that is causing things so try this:

$SPLUNK_HOME/bin/splunk btool list inputs --debug | grep lsof

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2 — Wed, 30 Sep 2020 02:42:47 GMT

I had to remove/re-install the splunk_TA_nix app to get it to stop behaving this way.

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

corybits — Thu, 04 Jun 2020 15:49:00 GMT

Thank you for the insights shared, my question is; when you set the indexes for all the Splunk_TA_nix inputs in Splunk_TA_nix/local/inputs.conf, did you use the same index for all the inputs, and was the index/indexes defined in your indexes.conf? Bare with me, new to Splunk.