What scripts are available to run against splunk log files to identify error situations

[lvirden]

There are such a variety of log files and I am uncertain what logs contain things that a splunk admin needs to address immediately. Are there scripts that have been developed to look against the current log files to determine concerning warning and error messages?

[jkat54]

I recommend fixing all errors and warnings found when you run this search:

index=_internal log_level=warn OR log_level=error

In distributed environments please make sure you are forwarding your internal indexes or run this search on every splunk instance.

I've seen simple warning messages bring down clusters, this is why I recommend fixing ALL errors AND warnings found with the above search. To my knowledge there arent any scripts for doing this, but the simple search above ran by a splunk admin should get you started in the right direction.

Also, there is a btool check command line tool that checks the syntax of your configuration files:
./splunk btool check

There are other "tools" / "apps" that help you figure this stuff out though. Most certainly you can get a good idea of indexing, forwarding and search health via the DMC. Also there is an app called S.o.S. (Splunk on Splunk), and another great app called fire brigade.

http://docs.splunk.com/Documentation/Splunk/6.4.3/DMC/DMCoverview
https://splunkbase.splunk.com/app/1632/
https://splunkbase.splunk.com/app/748/

[lvirden]

These sound like great starting points. I'm so new to splunk that I'm untrained at this point - just trying to grind through the log to help locate an odd behavior we started seeing over the weekend (a user reporting that he is no longer getting the reports he has set up).

I am hoping to gather information from the logs to identify some starting places (rebooting, remounting, etc)

Thank you very much. I'm a bit surprised that there are not a lot of scripts written to admin the tool. Most Unix vendor software I've used tended to either come with useful scripts or the community around the software had developed useful scripts.

Much to learn! Thank you for the pointers.

[jkat54]

In this case you use the tool yourself to discover and solve issues. The community is vibrant and helps you with just about anything. In some cases I write custom code just for folks who need to solve very specific problems. However it would be a disservice to just point you at a github and tell you to "run these scripts". If you have an issue you come here and let us know... We'll point you in the right direction for sure. It might be a script, we might tell you to call support, we might tell you to check out an app for Splunk, etc. Everyone has unique situations and setups so there isn't a magic bullet here.