How does Splunk decide on which splunkd WARNs are important enough to be displayed in the blue warning bar and which are filtered out?
For example the following message will get displayed:
01-11-2011 12:16:51.012 WARN databasePartitionPolicy - applying indexing throttle for <indexPath> because bucket has too many tsidx files, is your splunk-optimize working?
but this one won't:
01-11-2011 13:45:41.474 WARN databasePartitionPolicy - Handling shutdown or signal in DatabasePartitionPolicy: <indexPath>
There should be some kind of a default list of filters...
Rational - I want to create a scheduled search that will email the "important" warnings to splunk admin.
The importance of messages are hardcoded, so the decision is made by the developer.
There is no list of all error messages you could use to build a lookup, so you may want to start with a simple count
ERROR / WARN / FATAL events per component.
index=_internal source=*splunkd.log* NOT log_level=INFO NOT log_level=DEBUG
| chart count by component log_level
or check the dashboards on the SOS app.