Alerting

Report, Alert and suggest action on Splunk internal errors

bkumarm
Contributor

Splunk admins are facing regular challenge in understanding the error message and what corrective action need to be taken? or just ignore the internal errors.
sometimes these error messages are huge and may need cleanup or marking them for someone else's notice.

Example logs:
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig is obsolete! For replacement check ip.
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Because testband address has 20 bytes, only the first 8 bytes are displayed correctly.
02-26-2018 05:04:49.871 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
02-26-2018 00:01:25.118 -0500 ERROR FrameworkUtils - Incorrect path to script: /.binrunshell.cmd. Script must be located inside $SPLUNK_HOME/bin/scripts.
02-26-2018 00:01:25.118 -0500 ERROR ExecProcessor - Ignoring: "/.binrunshell.cmd mymsg"
02-26-2018 00:01:13.717 -0500 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /output_mode=json: Could not find object id=asdflkj

I would like to combine all the Execprocessor say ignore.
All FrameworkUtils error as send email to developer.

Any idea on how can we extract the error message part from these events and correlate( group) them ? I need help in getting the regex for doing it.
Any other approach is also welcome.

0 Karma

valiquet
Contributor

index=_internal sourcetype=splunkd log_level!=INFO |cluster

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Rather than ignore the ExecProcessor errors, disable the scripts to keep them from running and producing noise in your logs.

---
If this reply helps you, Karma would be appreciated.
0 Karma

bkumarm
Contributor

Yes, that would be final solution. we want to catch all such errors and alert user.
There are apps being deployed by large number of users. Hence we need a generic approach.
The idea is to first - parse the event and get the message part. secondly, check if the message with the one we have in a lookup table. Third step generate alert message and initiate corrective action.

0 Karma

bkumarm
Contributor

My problem is, I am unable to get the transforms to extract the message body, in such a way that I can map it into a lookup table field.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Why transforms? Does it have to be done at index time?
For a search-time extraction, try ... | rex "(?:ExecProcessor|FrameworkUtils) - (?<Message>.*)" | ....
What does your lookup table look like?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...