Alerting

Report, Alert and suggest action on Splunk internal errors

Contributor

Splunk admins are facing regular challenge in understanding the error message and what corrective action need to be taken? or just ignore the internal errors.
sometimes these error messages are huge and may need cleanup or marking them for someone else's notice.

Example logs:
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig is obsolete! For replacement check ip.
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Because testband address has 20 bytes, only the first 8 bytes are displayed correctly.
02-26-2018 05:04:49.871 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
02-26-2018 00:01:25.118 -0500 ERROR FrameworkUtils - Incorrect path to script: /.binrunshell.cmd. Script must be located inside $SPLUNKHOME/bin/scripts.
02-26-2018 00:01:25.118 -0500 ERROR ExecProcessor - Ignoring: "/.binrunshell.cmd mymsg"
02-26-2018 00:01:13.717 -0500 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /output
mode=json: Could not find object id=asdflkj

I would like to combine all the Execprocessor say ignore.
All FrameworkUtils error as send email to developer.

Any idea on how can we extract the error message part from these events and correlate( group) them ? I need help in getting the regex for doing it.
Any other approach is also welcome.

0 Karma

Contributor

index=internal sourcetype=splunkd loglevel!=INFO |cluster

0 Karma

SplunkTrust
SplunkTrust

Rather than ignore the ExecProcessor errors, disable the scripts to keep them from running and producing noise in your logs.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Contributor

Yes, that would be final solution. we want to catch all such errors and alert user.
There are apps being deployed by large number of users. Hence we need a generic approach.
The idea is to first - parse the event and get the message part. secondly, check if the message with the one we have in a lookup table. Third step generate alert message and initiate corrective action.

0 Karma

Contributor

My problem is, I am unable to get the transforms to extract the message body, in such a way that I can map it into a lookup table field.

0 Karma

SplunkTrust
SplunkTrust

Why transforms? Does it have to be done at index time?
For a search-time extraction, try ... | rex "(?:ExecProcessor|FrameworkUtils) - (?<Message>.*)" | ....
What does your lookup table look like?

---
If this reply helps you, an upvote would be appreciated.
0 Karma