We recently encountered an issue with a search that returned this error message:
[subsearch]: File '/opt/splunk/var/run/splunk/file_name.csv' could not be opened for reading.
While the fix was fairly simple, we would like to be notified when this type of issue starts in the future. I checked all of the index="_*" logs, and I do not see the error message. Is there a way for me to enable logging for this type of problem?
Have you tried looking at the S.o.S. App (Splunk on Splunk)... this is a diagnostics App for your Splunk installation there are a lot of useful searches/views here.
The S.o.S app can be found here... http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk
This is an App created by Splunk themselves, and is very helpful for multiple troubleshooting tools. It builds on the information from your installation (conf files, logs, etc.).
The SOS App uses logs in the internal index, which I was searching before. However, when I looked again with a looser search, I found the cause - but not the error seen in the web browser. I'll play around with alerting based on loglevel.
Good luck with this. I don't think you will always see the same error as in the logs... normally the logs would be for a "techie" and the web errors in a more human readable manner (for the general user)... As you probably already know 😛