Alerting

Trying to create a custom alert action passing a host value to the script that runs a linux command with that value as an argument

mjones414
Contributor

In this case I'm using a PBS job scheduler and whenever splunk sees a uncorrectable memory error I want it to offline the node within PBS itself.

the local command line syntax would be something like:

pbsnodes -o $hostname$
qmgr -c 's n $hostname$ comment="Splunk offlined this node due to uncorrectable memory errors"

Thus far I've been unsuccessful in having any way to trigger this as an alert action to a search.

Any help would be greatly appreciated!

0 Karma

harsmarvania57
Ultra Champion

Can you please provide more information on Custom Alert Action ? Have you created Custom Alert Action ? If yes, then is it possible you to provide your script which is running above command and other configuration file ?

If you are running bash/shell script in Custom Alert Action then have a look at answers thread on https://answers.splunk.com/answers/734938/custom-alerts-how-to-use-configured-variables-and-1.html , you will get idea how to read results of splunk query and then perform action on each output event in your script.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...