Why is lsof_sos.sh not returning any data?

[Jason]

We have just deployed TA-sos to all search heads and indexers. Both inputs (ps_sos.sh and lsof_sos.sh) are enabled, but no lsof_sos source data is being received. Running the script manually, it produces no output. Any idea what is going wrong?

lsof is not on path, but even editing the script to call lsof at its actual location /usr/sbin/lsof still produces no output.

/usr/sbin/lsof -n -P -s -p [splunkd pid from ./splunk status],[splunkweb pid from ./splunk status] produces plenty of output, even run as the splunk user.

[asimagu]

it may be a problem of $SPLUNK_HOME not having been set, therefore when the scripts tries to find that variable, it is not able to find it

[heybigben]

UPDATE: This is fixed in S.o.S 3.1 and will be fixed in the next release of the S.o.S addon for Linux and Unix (2.0.5, in all likelihood).

We had the same issue where lsof data was not showing up in splunk. This problem was happening in both the unix app and the SOS app despite the inputs being enabled. On our rhel5.8 system lsof is at /usr/sbin/lsof but both splunk apps - in common.sh for linux systems only set the path to /sbin which resulted in the lsof command not being found and thus no data was being returned. As a temp workaround I set common.sh to have the following for the path and lsof data started showing up and the graphs generated. PATH=$PATH:/sbin/:/usr/sbin/ We will be submitting a ticket to splunk support soon to get this fixed or see if there is a better solution.

[hexx]

We expect to put out a new version including this fix in the next few weeks.

[yoho]

Hi ! This problem still exists. It's easy to fix if you have a deployment server but I think apps should work correctly by default without any tuning, so waiting desperately for a 2.0.5 release of TA-sos...

[hexx]

Thanks for sharing your investigation, @heybigben! We'll assess your findings within the context of the bug currently filed against this issue (SUP-649).
If anyone else experiencing this problem is able to resolve it using @heybigben's work-around, I'd love to hear about it.

[hexx]

Thank you for the information provided. I have opened a bug against the S.o.S app (internal reference: SUP-649) to have this investigated and fixed. Hopefully, we can easily reproduce this in-house.

[jrodman]

answers is pretty awkward for debugging; but i would try (whichever the script asks for, bash or sh)

$ bash -x -v lsof_sos.sh

If you're in an fairly sensitive environment you may want to do some quick greps on the otput -- maybe hostnames may show up.

[hexx]

And just to be clear: Does running /usr/sbin/lsof -n -P -s -p <splunkd PID> manually as the splunk user yield output as expected?

[Jason]

Thanks Laks. Yes Hexx Splunk is running as the splunk user. It's running on Oracle enterprise linux, apparently a rebadged RHEL 5.6.

[lakshman237]

output the other command.

[splunk@dyl10639app21 bin]$ /opt/splunk/bin/splunk cmd /opt/splunk/etc/apps/sos/bin/lsof_sos.sh
[splunk@dyl10639app21 bin]$ cd /opt/splunk/etc/apps/sos/bin/
[splunk@dyl10639app21 bin]$ ./lsof_sos.sh

sorry had to send the results in comment and in answer as i am unable to post more. I work with Jason on this .

[lakshman237]

[splunk@dyl10639app21 bin]$ /usr/sbin/lsof -v
lsof version information:
revision: 4.78
latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
latest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_man
constructed: Wed Jun 6 05:06:54 EDT 2007
constructed by and on: mockbuild@ca-build14
compiler: cc
compiler version: 4.1.1 20070105 (Red Hat 4.1.1-52)
compiler flags: -DLINUXV=26016 -DGLIBCV=205 -DHASIPv6 -DHASSELINUX -D_FILE_OFFSET_BITS=64 -DLSOF_VSTR="2.6.16" -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
loader flags: -L./lib -llsof -lselinux
system info: Linux ca-build14 2.6.20-1.2952.fc6 #1 SMP Wed May 16 18:18:22 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
Anyone can list all files.
/dev warnings are disabled.
Kernel ID check is disabled.

[lakshman237]

Hi, its Oracle Enterprise Linux
uname -a
Linux dyl10639app21 2.6.18-238.el5 #1 SMP Tue Jan 4 15:41:11 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

[hexx]

Hi, Jason. Could you tell us what OS and distribution this is running on? Also, could you provide the output of the following commands?

# /usr/sbin/lsof -v
# $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/etc/apps/sos/bin/lsof_sos.sh

Also, is splunkd running as root or as a dedicated user?