topic Re: Report, Alert and suggest action on Splunk internal errors in Alerting

Report, Alert and suggest action on Splunk internal errors

bkumarm — Tue, 29 Sep 2020 18:11:58 GMT

Splunk admins are facing regular challenge in understanding the error message and what corrective action need to be taken? or just ignore the internal errors.
sometimes these error messages are huge and may need cleanup or marking them for someone else's notice.

Example logs:
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig is obsolete! For replacement check ip.
02-26-2018 05:04:49.872 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Because testband address has 20 bytes, only the first 8 bytes are displayed correctly.
02-26-2018 05:04:49.871 +0000 ERROR ExecProcessor - message from "/scripts/test.sh" Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
02-26-2018 00:01:25.118 -0500 ERROR FrameworkUtils - Incorrect path to script: /.binrunshell.cmd. Script must be located inside $SPLUNK_HOME/bin/scripts.
02-26-2018 00:01:25.118 -0500 ERROR ExecProcessor - Ignoring: "/.binrunshell.cmd mymsg"
02-26-2018 00:01:13.717 -0500 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /output_mode=json: Could not find object id=asdflkj

I would like to combine all the Execprocessor say ignore.
All FrameworkUtils error as send email to developer.

Any idea on how can we extract the error message part from these events and correlate( group) them ? I need help in getting the regex for doing it.
Any other approach is also welcome.

Re: Report, Alert and suggest action on Splunk internal errors

richgalloway — Mon, 26 Feb 2018 13:48:49 GMT

Rather than ignore the ExecProcessor errors, disable the scripts to keep them from running and producing noise in your logs.

Re: Report, Alert and suggest action on Splunk internal errors

bkumarm — Tue, 27 Feb 2018 05:51:33 GMT

Yes, that would be final solution. we want to catch all such errors and alert user.
There are apps being deployed by large number of users. Hence we need a generic approach.
The idea is to first - parse the event and get the message part. secondly, check if the message with the one we have in a lookup table. Third step generate alert message and initiate corrective action.

Re: Report, Alert and suggest action on Splunk internal errors

bkumarm — Tue, 27 Feb 2018 05:52:22 GMT

My problem is, I am unable to get the transforms to extract the message body, in such a way that I can map it into a lookup table field.

Re: Report, Alert and suggest action on Splunk internal errors

richgalloway — Tue, 27 Feb 2018 17:15:26 GMT

Why transforms? Does it have to be done at index time?
For a search-time extraction, try ... | rex "(?:ExecProcessor|FrameworkUtils) - (?<Message>.*)" | ....
What does your lookup table look like?

Re: Report, Alert and suggest action on Splunk internal errors

valiquet — Tue, 29 Sep 2020 18:21:30 GMT

index=_internal sourcetype=splunkd log_level!=INFO |cluster