All Apps and Add-ons
Highlighted

How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

I can't figure out why lsof.sh is running every minute.
Here's the
"btool inputs list --debug" output for lsof:

/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = 1
/opt/splunkforwarder/etc/system/local/inputs.conf host = c20sbap01l01
/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf index = cre_linux
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf interval = 600
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf source = lsof
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf sourcetype = lsof

Here's my splund.log output:

10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh
10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - interval: 60000 ms

I've tried restarting splunk to no effect...
Notice that the interval is set to 600 (600 seconds) in the btool output, but 60000 (60 seconds) in the splunkd.log output.
I'll try interval = -1 next, and then a single app after that.

Labels (1)
0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

SplunkTrust
SplunkTrust

Do you want to stop lsof.sh from running at all or just make it run every 10 minutes?

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

stop it altogether. some of my servers have 5M files open at a time...

0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Motivator

Dumb question, but did you run btool on the same machine that the splunkd log is from?

Cheers,
Jacob
0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

Yes, I did. I have since stopped Splunk, removed deploymentclient.conf, and the DS2-SplunkTAnix-cre directory, and set the indexes for all the SplunkTAnix inputs in SplunkTAnix/local/inputs.conf, and restarted splunk. I'm still getting a steady stream of lsof events every minute.

0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

This is on Splunk Universal Forwarder 7.0.1.

0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

Ok, I stopped Splunk, removed the SplunkTAnix app, started splunk put the app back and started splunk, and I'm finally no longer getting lsof events. However, I now need to do the same on all my deployment clients... Good thing I was planning on working late.

0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

I should say I restarted Splunk after I put the app back.

0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Esteemed Legend

The btool is your friend here. You could have an inputs.conf in any app that is causing things so try this:

$SPLUNK_HOME/bin/splunk btool list inputs --debug | grep lsof
0 Karma
Highlighted

Re: How can I get Splunk_TA_nix to stop running lsof.sh?

Path Finder

I had to remove/re-install the splunkTAnix app to get it to stop behaving this way.

View solution in original post

0 Karma