Alerting

Report, Alert and suggest action on Splunk internal errors

bkumarm
Contributor

Splunk admins are facing regular challenge in understanding the error message and what corrective action need to be taken? or just ignore the internal errors.
sometimes these error messages are huge and may need cleanup or marking them for someone else's notice.

Example logs:
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig is obsolete! For replacement check ip.
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Because testband address has 20 bytes, only the first 8 bytes are displayed correctly.
02-26-2018 05:04:49.871 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
02-26-2018 00:01:25.118 -0500 ERROR FrameworkUtils - Incorrect path to script: /.binrunshell.cmd. Script must be located inside $SPLUNK_HOME/bin/scripts.
02-26-2018 00:01:25.118 -0500 ERROR ExecProcessor - Ignoring: "/.binrunshell.cmd mymsg"
02-26-2018 00:01:13.717 -0500 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /output_mode=json: Could not find object id=asdflkj

I would like to combine all the Execprocessor say ignore.
All FrameworkUtils error as send email to developer.

Any idea on how can we extract the error message part from these events and correlate( group) them ? I need help in getting the regex for doing it.
Any other approach is also welcome.

0 Karma

valiquet
Contributor

index=_internal sourcetype=splunkd log_level!=INFO |cluster

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Rather than ignore the ExecProcessor errors, disable the scripts to keep them from running and producing noise in your logs.

---
If this reply helps you, Karma would be appreciated.
0 Karma

bkumarm
Contributor

Yes, that would be final solution. we want to catch all such errors and alert user.
There are apps being deployed by large number of users. Hence we need a generic approach.
The idea is to first - parse the event and get the message part. secondly, check if the message with the one we have in a lookup table. Third step generate alert message and initiate corrective action.

0 Karma

bkumarm
Contributor

My problem is, I am unable to get the transforms to extract the message body, in such a way that I can map it into a lookup table field.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why transforms? Does it have to be done at index time?
For a search-time extraction, try ... | rex "(?:ExecProcessor|FrameworkUtils) - (?<Message>.*)" | ....
What does your lookup table look like?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Get Inspired! We’ve Got Validation that Your Hard Work is Paying Off

We love our Splunk Community and want you to feel inspired by all your hard work! Eric Fusilero, our VP of ...

What's New in Splunk Enterprise 9.4: Features to Power Your Digital Resilience

Hey Splunky People! We are excited to share the latest updates in Splunk Enterprise 9.4. In this release we ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...