Alerting

How to restart a remote Windows service from a Splunk search alert script?

rainhailrob
Path Finder

I want to restart a remote Windows service from a Splunk search alert script. How do I pass the server name to the script? Is it possible using PowerShell? Do I need to use the same service account running the splunkforwarder service on the client as is running on the SPlunk search head.

0 Karma

pjamesburwell
Engager

INTERESTING. But the call is the issue (cannot do PS calls either). No way to tell the UF system to launch the script locally after a head alert without a UAC override through Splunkd rights elevation? I guess Phantom is it then.
Thank you!

0 Karma

pjamesburwell
Engager

NO. How do you USE SPLUNK to restart a service running on a remote server in the network?,I don't want to restart Splunk Enterprise, I want SPLUNK ENTERPRISE to restart a remote service.
I want to set an event for a specific app Service, running on a VMWARE 5 Microsoft Windows 2012R2 Guest, we are monitoring with Splunk to restart the application service (Image Gateway) when the logfile shows the service went to sleep (due to fax server comm loss).

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi pjamesburwell,

you might want to check the alert action section in the docs https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsIntro to read what is possible. The most important question would be how you can restart the service remotely and how it can be scripted.

Another option would be to check out https://www.splunk.com/en_us/software/splunk-security-orchestration-and-automation.html which is purpose built for such use cases.

Hope this helps ...

cheers, MuS

0 Karma

pjamesburwell
Engager

Thank you, I will take a look...
But specifically I am trying to determine HOW Splunk restarts an application service (NOT a SPLUNK service) from a forwarder.
Does it do an RPC call?
Does it use Powershell?
Does it use the Incoming port 8089?

0 Karma

MuS
SplunkTrust
SplunkTrust

Happy new year pjamesburwell,

As I wrote before The most important question would be how you can restart the service remotely and how it can be scripted.
So, If you can restart the service using RCP, use RCP - If you can restart the service using Powershell, use Powershell.
No, it does not use the Splunk management port 8089 - this port is only for Splunk internal activities.
Another thing I just noticed, you will not be able (as far as I know) to do such a thing on the forwarder. Your alert must run on a full Splunk enterprise instance and that instance will use the remote service restart script to remotely restart the service.

Hope this helps ...

cheers, MuS

0 Karma

pjamesburwell
Engager

By "remote service restart script" do you mean the Restart-Service command from the module Microsoft.PowerShell.Management [https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/restart-service?v...]? If that's the case then we are headed the right way. We are going to put a script like this in the Universal Forwarder "c:\Program Files\SplunkuniversalFowarder\bin\scripts" folder:

(Alert that sets variable)
Get-Module -ListAvailable Microsoft*| Select name | Out-String | % {$_ -replace "Name"}|% {$_ -replace "-"}|% {$_ -replace " "} #|out-file -filepath .\ModulesbyName_Microsoft.txt
Wait-Event -Timeout 5
Get-Content -Path C:\scripts\ModulesbyName_Microsoft.txt | Import-Module -verbose
Wait-Event -Timeout 5.1
return (Restart-Service -name "NameofService" -ErrorAction SilentlyContinue),$?
(Splunk script to take alert and fwd to eventlog)

-OR-

After manual update to PS5.1

Get-Service -Name "PartialNameofService*" | Where-Object {$_.Status -eq "Stopped"} | Restart-Service
Get-Service -Name "PartialNameofService*" | Where-Object {$_.Status -eq "Paused"} | Restart-Service

-OR-

After manual update to PS5

A simple specific command called from Splunk Alert - Other alerts set to watch for EventLog entries

Get-Service -Name "PartialNameofService*" | Restart-Service
0 Karma

nickhills
Ultra Champion

Restarting a service will have nothing to do with a universal forwarder - you don't need any scripts on the UF.

You need to devise a method of triggering a restart outside of Splunk - perhaps you write a powershell script which takes a target host name, and then executes the script with that host as a target.

What @MuS is getting at is that restarting a service happens outside of Splunk - It can certainly be done, but the mechanism as to how its done is in your gift alone - not Splunks.

Once you know how you are going to restart the service you can build an alert on your Splunk Search head to trigger the script against the target server - but the important bit to note is that it's the Splunk server which will execute the restart (with all of the complexities of authentication and error handling) - NOT the UF.

If you need something more 'off-the-shelf' you will need something like Phantom which is designed for this

If my comment helps, please give it a thumbs up!
0 Karma

pjamesburwell
Engager

Then WHY does the "c:\program files\SplunkUniversalForwarder\bin\scripts" folder have a README that says, "Scripts placed in this directory can be called by Alerts for execution",?alt text

0 Karma

nickhills
Ultra Champion

Without meaning to sound discourteous to the good folks at Splunk, I’d probably suggest oversight.
It’s because that file is included in the barebones template on which many splunk apps are built upon. In the context of a UF app, it is perhaps misleading since alerts are not generated on/by UFs.

If my comment helps, please give it a thumbs up!
0 Karma

MuS
SplunkTrust
SplunkTrust

Look @pjamesburwell a lot of us on answers are Splunk users, been through almost every possible use case, and just try to be helpful by providing information or answers.

There is no way for a universal forwarder to trigger an alert action on its own. BUT, and this is me being pedantic here 😉 you could place the alert action script into c:\program files\SplunkUniversalForwarder\bin\scripts and call it directly from the search head using UNC path names (given you can connect from the search head to the server running the universal forwarder, and the script is allowed to restart a service).

You see, again the restart is done outside of Splunk like @nickhillscpl perfectly explained.

If you need a more solid alternative to this, have a look at Phantom https://www.splunk.com/en_us/software/splunk-security-orchestration-and-automation.html which is a tool for orchestration and automation and could do this.

cheers, MuS

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi rainhailrob,

to answer your question, you cannot pass the host name directly to the script. But you can pass some other arguments to the script, like http://docs.splunk.com/Documentation/Splunk/6.2.3/Alert/Configuringscriptedalerts :

Arg  Env Variable   Value
0   SPLUNK_ARG_0    Script name
1   SPLUNK_ARG_1    Number of events returned
2   SPLUNK_ARG_2    Search terms
3   SPLUNK_ARG_3    Fully qualified query string
4   SPLUNK_ARG_4    Name of report
5   SPLUNK_ARG_5    Trigger reason
6   SPLUNK_ARG_6    Browser URL to view the report.
7   SPLUNK_ARG_7    Not used for historical reasons.
8   SPLUNK_ARG_8    File in which the results for the search are stored. Contains raw results.

The last one is the best one to use in your script and read the host from the result and do what ever needed to be done with it.

Hope that helps ...

cheers, MuS

pjamesburwell
Engager

This is NOT the answer

0 Karma

MuS
SplunkTrust
SplunkTrust

The OP have asked How do I pass the server name to the script? so how can this not be the answer to this question?

cheers, MuS

0 Karma

sunrise
Contributor
0 Karma

pjamesburwell
Engager

This is a question about restarting a service WITH Splunk.

0 Karma

rainhailrob
Path Finder

I had tried passing parameters to the script, but I was unsuccessful.

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...