Alerting

Need some assistance in how to trouble shoot a Modular Alert

snappersdad
New Member

All,

I have tried to create a Modular Alert, running under Splunk 7.6.2, based on the blog by Luke Murphy: https://www.splunk.com/blog/2016/08/22/how-to-create-a-modular-alert.html.

I thought I had thoroughly followed the instructions and indeed I can see in the search results where the custom action appears to run successfully:

Time    Event

10/28/19 10-28-2019 12:10:00.649 -0400 INFO sendmodalert - action=make_a_log_message - Alert action script completed in duration=332 ms with exit code=0

10/28/19 10-28-2019 12:10:00.315 -0400 INFO sendmodalert - Invoking modular alert action=make_a_log_message for search="Modular Alert" sid="scheduler_adminsearch_RMD5a82eb824001bf163_at_1572279000.1 " in app

However, none of the other expected logging is happening.

None of the troubleshooting techniques I could think of as well as read about, have provided any insights to what I am doing wrong...

Any help anyone can provide as to how to troubleshoot, would be greatly appreciated.

Thanks,

0 Karma

LukeMurphey
Champion

Do you see results when you run a search for the following?

index="_internal" "sendmodalert - action=make_a_log_message"

0 Karma

snappersdad
New Member

No, nothing...

None of the print statements I put in, no matter where they were, generated any output at all.

The only time I saw any output was when I changed the format of the print statement.

0 Karma

snappersdad
New Member

So, I ended up attempting to run the Python script standalone:

I'm running under Debian Linux so the command looked like this:

cd /opt/splunk
sudo bin/splunk cmd python etc/apps/modular_alert_example/bin/make_a_log_message.py

and quickly realized that the Python script I was running was written for Python 3. However, my environment is Python 2.7.13.

So, I left the directory structure, and everything else in place and simply replaced the make_a_log_message.py script with a simple "Hello World" script written to a file.

And that was successful!

He is a copy of the test script (for anyone else who might have a need for something like this)

from future import print_function
import sys, os, datetime

f = open(os.path.join("/opt/splunk", "var", "log", "splunk", "test_modalert.log"), "a")
print(str(datetime.datetime.now().isoformat()), "Hello World", file=f)
f.close()

and here is a copy of the test file contents:

2019-10-29T14:38:49.213920 Hello World
2019-10-29T14:52:00.973576 Hello World

Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...