Getting Data In

Issues Triggering Alert Scripts on Windows XP

maverick
Splunk Employee
Splunk Employee

I’m currently running Splunk on my Windows XP SP3 and I'm trying to get a couple scripts to run after an alert triggers, but failing all the while. Not sure what I could be doing wrong.

The environment variable for SPLUNK_HOME = C:\Program Files\Splunk

Scrpt #1
Name: echo.cmd
Location: C:\Program Files\Splunk\bin\scripts
Filename of shell script to execute: echo.cmd (from the Saved Search pop-up dialog box)

Scrpt #2
Name: sendtrap.pl
Location: C:\Program Files\Splunk\bin\scripts
Filename of shell script to execute: sendtrap.pl (from the Saved Search pop-up dialog box)

Neither of these currently get triggered at all.

One thing that I have wondered is whether Splunk may be having an issue with the space between “Program” and “Files” in the SPLUNK_HOME environment variable.

Also, in my Perl script, I correctly reference the library (as shown below), per the online Splunk docs regarding this topic.


#!C:\Perl\bin\perl
#
# sendsnmptrap.pl: A script to for Splunk alerts #


Any help or insight would be greatly appreciated.

1 Solution

hexx
Splunk Employee
Splunk Employee

This can be a complex problem and it's important to be thorough in checking that every step of the process (from scheduled search to alert script) is working as expected :

A) Is my scheduled search running?

  • Check %SPLUNK_HOME%\var\log\splunk\scheduler.log or search for "index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by status" to determine if your scheduled search is running and with a status of "success"? If you see failures here, drill down into those to see why the search is not running. Is it taking too long to execute? Are there too many concurrent searches running at that time?

B) Is my scheduled search generating the expected results?

  • Again, check %SPLUNK_HOME%\var\log\splunk\scheduler.log or search for "index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by result_count". Is the result event count as you would expect it?

  • Provided you have configured Splunk to be able to send emails, add an email action to send yourself an email . Check the search results in the URL provided by the email, and see if they are as expected.

C) Is my alert action being triggered?

  • Set up an additional alert action (typically an email) to see if alert actions are being triggered or not.

  • Make sure that you don't have an issue with the condition of your alert. To verify this, change the alert condition to "always". If your script runs then, you know the problem is with the condition and you should study the results of your scheduled search to see why it isn't triggering the alert action as expected.

D) Is my alert script working?

  • First, make sure that your script sits where it should : %SPLUNK_HOME%\bin\scripts\ is a good location, but you may also want to put it in %SPLUNK_HOME%\etc\apps\your_app_name\bin\ if your scheduled search is app-specific and not global.

  • Let's check that the script itself runs outside of Splunk. As the user that splunkd runs as on your system, launch the script. Is it producing the expected output? If the script is dependent on variables passed by the Splunk scheduled search, you may want to temporarily set those to hard values in the script itself.

  • Use the "runshellscript " command to run your script manually from the search bar. Does this work?

  • If your own script is running manually and from the search bar but not when called as an alert action, it's time to check if Splunk is able to run a simple script in that manner. Splunk ships with a simple script located in %SPLUNK_HOME%\bin\scripts called echo.bat (or echo.sh for *nix systems). This script does a very simple thing when called : It attempts to write the 8 arguments passed by the Splunk scheduled search to the script (see http://www.splunk.com/base/Documentation/latest/Admin/Configurescriptedalerts#Script_options) to a text control file.

    This is a good script to test the basic functionality of triggering an alert script, and seeing how Splunk is passing variables to it. Note that you may want to change the script slightly, for example to write it's output to a different location or to add a time stamp to the output.

    I typically modify it in the following way in order to time stamp the output and also to specify an absolute path to a custom directory :

    @echo off
    echo %0, %1, %2, %3, %4, %5, %6, %7, %8 >> "c:\temp\echo_output.txt"
    date /T >> "c:\temp\echo_output.txt"
    time /t >> "c:\temp\echo_output.txt"
    echo ---------------------------------------- >> "c:\temp\echo_output.txt"

    To use this script, first run it manually as the user that runs splunkd and make sure that you get the expected result : A file called "echo_output.txt" should be created in %SPLUNK_HOME%\bin\scripts. If you have passed arguments to the script from the command line, they should be listed in the last line that was written to that file.

    Next, change your scheduled search to use echo.(bat|sh) instead of your own script, and observe the control file (echo_output.txt or any other file you may have pointed to script to output to).

    If none of the above steps allow you to solve your problem, it's probably time to contact Splunk Support!

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You can also consult Perl docs to see how they suggest making the association, or http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ftype.mspx?mfr=true

Though, again, in this case, you don't need to do this, and I would recommend against it.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I will note that the #! stuff to trigger perl has no effect on Windows, so you might as well skip it. You have to use ASSOC and/or FTYPE (or Windows Explorer dialogs) to associate the .pl file with Perl, or wrap the call in a .cmd script, or simply use the Windows CMD version of that script (which itself is just a wrapper around the net-snmp sendsnmptrap.exe program anyway).

0 Karma

hexx
Splunk Employee
Splunk Employee

This can be a complex problem and it's important to be thorough in checking that every step of the process (from scheduled search to alert script) is working as expected :

A) Is my scheduled search running?

  • Check %SPLUNK_HOME%\var\log\splunk\scheduler.log or search for "index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by status" to determine if your scheduled search is running and with a status of "success"? If you see failures here, drill down into those to see why the search is not running. Is it taking too long to execute? Are there too many concurrent searches running at that time?

B) Is my scheduled search generating the expected results?

  • Again, check %SPLUNK_HOME%\var\log\splunk\scheduler.log or search for "index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by result_count". Is the result event count as you would expect it?

  • Provided you have configured Splunk to be able to send emails, add an email action to send yourself an email . Check the search results in the URL provided by the email, and see if they are as expected.

C) Is my alert action being triggered?

  • Set up an additional alert action (typically an email) to see if alert actions are being triggered or not.

  • Make sure that you don't have an issue with the condition of your alert. To verify this, change the alert condition to "always". If your script runs then, you know the problem is with the condition and you should study the results of your scheduled search to see why it isn't triggering the alert action as expected.

D) Is my alert script working?

  • First, make sure that your script sits where it should : %SPLUNK_HOME%\bin\scripts\ is a good location, but you may also want to put it in %SPLUNK_HOME%\etc\apps\your_app_name\bin\ if your scheduled search is app-specific and not global.

  • Let's check that the script itself runs outside of Splunk. As the user that splunkd runs as on your system, launch the script. Is it producing the expected output? If the script is dependent on variables passed by the Splunk scheduled search, you may want to temporarily set those to hard values in the script itself.

  • Use the "runshellscript " command to run your script manually from the search bar. Does this work?

  • If your own script is running manually and from the search bar but not when called as an alert action, it's time to check if Splunk is able to run a simple script in that manner. Splunk ships with a simple script located in %SPLUNK_HOME%\bin\scripts called echo.bat (or echo.sh for *nix systems). This script does a very simple thing when called : It attempts to write the 8 arguments passed by the Splunk scheduled search to the script (see http://www.splunk.com/base/Documentation/latest/Admin/Configurescriptedalerts#Script_options) to a text control file.

    This is a good script to test the basic functionality of triggering an alert script, and seeing how Splunk is passing variables to it. Note that you may want to change the script slightly, for example to write it's output to a different location or to add a time stamp to the output.

    I typically modify it in the following way in order to time stamp the output and also to specify an absolute path to a custom directory :

    @echo off
    echo %0, %1, %2, %3, %4, %5, %6, %7, %8 >> "c:\temp\echo_output.txt"
    date /T >> "c:\temp\echo_output.txt"
    time /t >> "c:\temp\echo_output.txt"
    echo ---------------------------------------- >> "c:\temp\echo_output.txt"

    To use this script, first run it manually as the user that runs splunkd and make sure that you get the expected result : A file called "echo_output.txt" should be created in %SPLUNK_HOME%\bin\scripts. If you have passed arguments to the script from the command line, they should be listed in the last line that was written to that file.

    Next, change your scheduled search to use echo.(bat|sh) instead of your own script, and observe the control file (echo_output.txt or any other file you may have pointed to script to output to).

    If none of the above steps allow you to solve your problem, it's probably time to contact Splunk Support!

piebob
Splunk Employee
Splunk Employee

hexx is the best!!! ❤️

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...