I have a script that runs in an app on my forwarders every 12 hours. Or at least that's what it was doing until it abruptly stopped working on all forwarders on September 30th. The script is deployed using the deployment server, so all the forwarders have identical configurations. The inputs.conf file has five scripts configured and all work except for this one. Here is its inputs.conf entry: [script://splunk/etc/apps/all/bin/chknmon.sh] interval = 43200 # Run every 12 hours sourcetype = nmonchk source = script://./bin/chknmon.sh The path to the chknmon.sh script is correct: [/splunk/etc/apps/all/default]# ls -las /splunk/etc/apps/all/bin/chknmon.sh 4 -rwxr-xr-- 1 xyz abc 96 Oct 04 14:13 /splunk/etc/apps/all/bin/chknmon.sh Splunk runs as root, and the script is owned by root. Running the script manually produces correct output. Other scripts in the same app (and same paths/permissions) run just fine. This one was no exception until it abruptly stopped working a few days ago. Admittedly, I've been toying around with Splunk a bit, but I have no idea what I could have done that could have affected just this one script. I've tried bouncing both the indexer and the forwarders, but that didn't help. I'm running 4.1.4 on the forwarders and 4.1.5 on the indexer. Thanks! ... View more

Whenever I (admin user) make a customization to the search app in Splunk, such as a field extraction, Splunk saves the props.conf in /splunk/etc/users/username/search/local/props.conf I understand why it does that, but I find myself having to copy the props.conf file to /splunk/etc/apps/search/local/props.conf so that the other users can take advantage of my customizations. Then I have to bounce Splunk, etc... Is there a way for me to make changes on-the-fly on the app level rather than on the user level? I'm sure there's a way, and I think I read it somewhere, but I can't find it. Thanks! ... View more

This is a tricky one (or is it?)... I have indexed Splunk data that looks like this (using multikv): device_name host list(Select) list(Disk) difference vpath0 xyz 19072176 fscsi2/hdisk28 13409 19058767 fscsi3/hdisk56 I'd like to be alerted if the difference between the two "list(Select)" fields is greater than 5%. I know how to do the math on paper, but I can't figure out how to apply it to Splunk. Essentially, I need to divide the "difference" by value of the higher "list(Select)", then multiply by 100. I've been dabbling with this, but now I'm wondering if it's even possible. Thanks! ... View more

I have a chart in a dashboard that shows a graph of paging space usage across all of our hosts. Or at least that's what it's supposed to do. It only seems to show the graph for ten of our twenty-two hosts. How can I tell it to graph all the hosts? My query looks like this: <searchString>sourcetype="paging" | timechart avg(percent_used) by host</searchString> Thanks! ... View more