topic Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data in Getting Data In

Shell Script to Check if Multiple Servers are Correctly Forwarding Data

mmensch — Fri, 29 Jan 2016 19:57:15 GMT

Hi all,

I have a list of servers in a text file "servers.txt."

I am trying to create a shell script that will see if all of my servers are sending data to Splunk on a daily basis. I have some scripting knowledge and created a script to see if the server is up or down by simply pinging it. I want to take this one step further. I want to know when the Splunk Forwarder has stopped working or something along those lines.

Any help would be appreciated.

Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data

lycollicott — Fri, 29 Jan 2016 20:19:34 GMT

You could script a remote "splunk status"

$serverlist = get-content splunk80uf.list

foreach ($server in $serverlist)
{
$result = Invoke-WmiMethod -ComputerName $server -Class Win32_Process -Name Create -ArgumentList "cmd.exe /c e:\app\SplunkUniversalForwarder\bin\splunk.exe status >E:\app\status80uf.log" 
if ($result.ReturnValue -ne 0)
    {
       ##$exception = New-Object System.ComponentModel.Win32Exception([int]$result.ReturnValue)
        ##Write-Error "Error launching installer on computer ${server}: $($exception.Message)"
        echo Crap
    }
}

sleep 5

foreach ($server in $serverlist)
{
write-host "$server`t" -nonewline
Get-content \\$server\e$\app\status80uf.log
}

Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data

lycollicott — Fri, 29 Jan 2016 20:21:42 GMT

On Unix/Linux you could take a similar approach if you setup ssh keys.

ssh user1@server1 command1

Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data

aljohnson_splun — Fri, 29 Jan 2016 22:33:59 GMT

I think it would be a lot easier to verify the completion of data being forwarded by searching 1.) the index the data is being forwarded to, and 2.) the _internal index to verify connection. Then, if you don't see data in 1, you can check 2 to make sure the connection is alive, and if 1 & 2 are blank, then you can investigate further.

Splunk is actually really great at doing this, so I would suggest avoiding a scripting language or shell scripts if possible.

Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data

mmensch — Mon, 01 Feb 2016 17:10:57 GMT

I understand Splunk is great at doing this, but this does not scale to thousands of servers. I have a list of 250 servers currently and it will grow into the tens of thousands.

Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data

mmensch — Fri, 05 Feb 2016 14:18:50 GMT

This is not working for me.

Re: Shell Script to Check if Multiple Servers are Correctly Forwarding Data

lycollicott — Fri, 05 Feb 2016 15:06:15 GMT

What kind of errors or messages do you get?