Can I auto-install or deploy Splunk onto all my remote Windows servers?

[maverick]

I have a lot of remote Windows servers that I would like to install Splunk on. It will take forever to manually do this, so I was wondering if I could use Group Policy Objects in Active Directory to automate the exe push. Also, I have some Linux servers too I may want to include, so I'm wondering what other "push" solution(s) do people use to distribute binaries to servers in their IT environments.

[Shane]

So in the following configuration. Can I just copy it to notepad and save as a .bat file? and also how does the syntax work for the path following the
SPLUNK_MSI=%~dp0\network drive\path\splunk-4.1.5-85165-x86-release.msi
or is it
SPLUNK_MSI=\network drive\path\splunk-4.1.5-85165-x86-release.msi
Also, does this install the software so it's just a forwarder? I'm trying to figure a way to deploy this agent out to my Windows servers and they can forward to my Splunk server. How do I point the servers to my splunk server?

I hope all these questions make sense. I plan to use SCCM to deploy this bat file.

****START CONFIG BELOW****

setlocal
IF "%PROCESSOR_ARCHITECTURE%" == "AMD64" goto b64
IF "%PROCESSOR_ARCHITEW6432%" == "AMD64" goto b64
:b32
set SPLUNK_MSI=%~dp0\splunk-4.1.5-85165-x86-release.msi
goto endb6432

:b64
set SPLUNK_MSI=%~dp0\splunk-4.1.5-85165-x64-release.msi
:endb6432

if not defined ProgramFilesW6432 (
set LOC=%ProgramFiles%\Splunk
) else (
set LOC=%ProgramFilesW6432%\Splunk
)

msiexec.exe /i "%SPLUNK_MSI%" INSTALLDIR="%LOC%" LAUNCHSPLUNK=0 SPLUNK_APP="" /QUIET
xcopy "%~dp0\etc" "%LOC%\etc" /s /f /y
copy /y "%LOC%\etc\splunk-forwarder.license" "%LOC%\etc\splunk.license"
pushd "%LOC%\bin\"
splunk restart --accept-license --no-prompt --answer-yes

popd
endlocal

[treinke]

We have created a batch installer also. To solve our needs we needed to be able to set up different indexes, file integrity monitoring, and which parts of Splunk to turn on. This is done by asking the installer a series of questions. We named the script file SplunkInstaller.cmd because we are so original and creative. The .cmd extension was used so in Windows Server 2008 you can choose to run as an administrator. We bundle this script, the msi files for 32bit and 64bit, and PortQry.exe in to a self extracting executable that will extract to the Windows Temp directory (%WINDIR%\Temp\ or commonly found at C:\Windows\Temp).

PortQry can be downloaded from Microsoft (http://support.microsoft.com/kb/310099). This allows you to test to make sure the ports are open to the Splunk indexer.

@echo off
cls

REM ##################################################
REM ## Checking to make sure all the files are here ##
REM ##################################################
:FileCheck
set Splunk32=splunk-4.1.2-79191-x86-release.msi
set Splunk64=splunk-4.1.2-79191-x64-release.msi

IF NOT EXIST %WINDIR%\temp\PortQry.exe goto MISSINGFILES
IF EXIST %WINDIR%\temp\%Splunk32% goto SplunkLocation
IF NOT EXIST %WINDIR%\temp\%Splunk64% goto MISSINGFILES

:SplunkLocation
REM ######################################################################
REM ## Setting the location Splunk gets its management information from ##
REM ######################################################################
echo.
echo Setting up the Deployment Client
echo Please select the location that the server should be pointing
echo   1. Location01
echo   2. Location02
echo   3. Location03
choice /c:123

IF ERRORLEVEL == 3 set DNSLOCATION=splunk1.location01.com
IF ERRORLEVEL == 2 set DNSLOCATION=splunk1.location02.com
IF ERRORLEVEL == 1 set DNSLOCATION=splunk1.location03.com

REM #########################################################################
REM ## This will test to make sure the ports are open through the firewall ##
REM #########################################################################
:PortTesting
echo Testing if the ports are open
%WINDIR%\temp\portqry -n %DNSLOCATION% -e 8089 -q
IF %ERRORLEVEL% == 0 goto Continue
IF %ERRORLEVEL% == 99 goto CantResolve
goto NeedNetworking

:Continue
%WINDIR%\temp\portqry -n %DNSLOCATION% -e 9997 -q
IF %ERRORLEVEL% == 0 goto PortFinished
IF %ERRORLEVEL% == 99 goto CantResolve
goto NeedNetworking

:PortFinished
echo The ports look open.  Continueing to install.
goto InstallType

:CantResolve
cls
echo Unable to resolve the Splunk DNS addresses (%DNSLOCATION%).
echo Please verify that the DNS settings are correct
goto END

:NeedNetworking
cls
echo Looks like the ports are not open.  Please open a ticet to networking
echo to open the following ports: TCP:8089 and TCP:9997 bi-directional.
goto END

REM #####################################
REM ## Install, upgrade, or uninstall? ##
REM #####################################
:InstallType
Echo.
echo Removing the old versions of Splunk (if any)
msiexec /x {78d6f15d-2645-4c7a-a677-36e6b4ddbd5c} /passive
echo.
echo Removing any old files (if any)
IF EXIST rmdir /s /q "c:\program files\Splunk"
IF EXIST rmdir /s /q "c:\Program Files (x86)\Splunk"
goto INSTALLING

REM ###########################################
REM ## Configuring the MSI to install Splunk ##
REM ###########################################
:INSTALLING
echo.
echo Default option is inside ().  The default option will be selected if nothing is pressed in 15 seconds.
echo.
echo.
choice /C YN /T 15 /D N /N /M "Forward the Windows Application Event Log? [Y|(N)]: "
IF ERRORLEVEL == 2 set WINEVENTLOGAPPCHECK=0
IF ERRORLEVEL == 1 set WINEVENTLOGAPPCHECK=1

choice /C YN /T 15 /D Y /N /M "Forward the Windows Security Event Log? [(Y)|N]: "
IF ERRORLEVEL == 2 set WINEVENTLOGSECCHECK=0
IF ERRORLEVEL == 1 set WINEVENTLOGSECCHECK=1

choice /C YN /T 15 /D N /N /M "Forward the Windows System Event Log? [Y|(N)]: "
IF ERRORLEVEL == 2 set WINEVENTLOGSYSCHECK=0
IF ERRORLEVEL == 1 set WINEVENTLOGSYSCHECK=1

set WINEVENTLOGFWDCHECK=0
set WINEVENTLOGSETCHECK=0

choice /C YN /T 15 /D N /N /M "Monitor the User Registry? [Y|(N)]: "
IF ERRORLEVEL == 2 set REGISTRYCHECK_U=0
IF ERRORLEVEL == 1 set REGISTRYCHECK_U=1

choice /C YN /T 15 /D N /N /M "Monitor the User Baseline Registry? [Y|(N)]: "
IF ERRORLEVEL == 2 set REGISTRYCHECK_BASELINE_U=0
IF ERRORLEVEL == 1 set REGISTRYCHECK_BASELINE_U=1

choice /C YN /T 15 /D N /N /M "Monitor the Local Machine Registry? [Y|(N)]: "
IF ERRORLEVEL == 2 set REGISTRYCHECK_LM=0
IF ERRORLEVEL == 1 set REGISTRYCHECK_LM=1

choice /C YN /T 15 /D N /N /M "Monitor the Local Machine Baseline Registry? [Y|(N)]: "
IF ERRORLEVEL == 2 set REGISTRYCHECK_BASELINE_LM=0
IF ERRORLEVEL == 1 set REGISTRYCHECK_BASELINE_LM=1

choice /C YN /T 15 /D N /N /M "Monitor the CPU Time? [Y|(N)]: "
IF ERRORLEVEL == 2 set WMICHECK_CPUTIME=0
IF ERRORLEVEL == 1 set WMICHECK_CPUTIME=1

choice /C YN /T 15 /D N /N /M "Monitor the Local Disk? [Y|(N)]: "
IF ERRORLEVEL == 2 set WMICHECK_LOCALDISK=0
IF ERRORLEVEL == 1 set WMICHECK_LOCALDISK=1

choice /C YN /T 15 /D N /N /M "Monitor the Free Disk? [Y|(N)]: "
IF ERRORLEVEL == 2 set WMICHECK_FREEDISK=0
IF ERRORLEVEL == 1 set WMICHECK_FREEDISK=1

choice /C YN /T 15 /D N /N /M "Monitor the Memory? [Y|(N)]: "
IF ERRORLEVEL == 2 set WMICHECK_MEMORY=0
IF ERRORLEVEL == 1 set WMICHECK_MEMORY=1

SPLUNK_APP=SplunkLightForwarder
FORWARD_SERVER="%DNSLOCATION%:9997"
LAUNCHSPLUNK=1

goto OSTYPE

REM #########################
REM ## finding the OS type ##
REM #########################
:OSTYPE
cls

echo.
if (%PROCESSOR_ARCHITECTURE%)==(x86) Then OSType="32bit"
if (%PROCESSOR_ARCHITECTURE%)==(AMD64) Then OSType="64bit"
echo Looks like this system is a %OSType% OS
echo.
echo Please select the OS type:
echo   1. 32bit
echo   2. 64bit
choice /c:12

IF ERRORLEVEL == 2 goto 64bit
IF ERRORLEVEL == 1 goto 32bit

:32bit
Echo This is a 32bit OS
msiexec.exe /passive /norestart /i %WINDIR%\temp\%Splunk32% WINEVENTLOGAPPCHECK="%WINEVENTLOGAPPCHECK%" WINEVENTLOGSECCHECK="%WINEVENTLOGSECCHECK%" WINEVENTLOGSYSCHECK="%WINEVENTLOGSYSCHECK%" WINEVENTLOGFWDCHECK="%WINEVENTLOGFWDCHECK%" WINEVENTLOGSETCHECK="%WINEVENTLOGSETCHECK%" REGISTRYCHECK_U="%REGISTRYCHECK_U%" REGISTRYCHECK_BASELINE="%WREGISTRYCHECK_BASELINE_U%" REGISTRYCHECK_LM="%WREGISTRYCHECK_LM%" REGISTRYCHECK_BASELINE="%WREGISTRYCHECK_BASELINE_LM%" WMICHECK_CPUTIME="%WWMICHECK_CPUTIME%" WMICHECK_LOCALDISK="%WWMICHECK_LOCALDISK%" WMICHECK_FREEDISK="%WWMICHECK_FREEDISK%" WMICHECK_MEMORY="%WWMICHECK_MEMORY%" SPLUNK_APP=SplunkLightForwarder FORWARD_SERVER="%DNSLOCATION%:9997" LAUNCHSPLUNK=1
goto CONFIG

:64bit
Echo This is a 64bit OS
msiexec.exe /passive /norestart /i %WINDIR%\temp\%Splunk64% WINEVENTLOGAPPCHECK="%WINEVENTLOGAPPCHECK%" WINEVENTLOGSECCHECK="%WINEVENTLOGSECCHECK%" WINEVENTLOGSYSCHECK="%WINEVENTLOGSYSCHECK%" WINEVENTLOGFWDCHECK="%WINEVENTLOGFWDCHECK%" WINEVENTLOGSETCHECK="%WINEVENTLOGSETCHECK%" REGISTRYCHECK_U="%REGISTRYCHECK_U%" REGISTRYCHECK_BASELINE="%WREGISTRYCHECK_BASELINE_U%" REGISTRYCHECK_LM="%WREGISTRYCHECK_LM%" REGISTRYCHECK_BASELINE="%WREGISTRYCHECK_BASELINE_LM%" WMICHECK_CPUTIME="%WWMICHECK_CPUTIME%" WMICHECK_LOCALDISK="%WWMICHECK_LOCALDISK%" WMICHECK_FREEDISK="%WWMICHECK_FREEDISK%" WMICHECK_MEMORY="%WWMICHECK_MEMORY%" SPLUNK_APP=SplunkLightForwarder FORWARD_SERVER="%DNSLOCATION%:9997" LAUNCHSPLUNK=1
goto CONFIG

REM ########################################################
REM ## Configure the Splunk agent to point to the indexer ##
REM ########################################################

:CONFIG
cls
echo.
echo Configuring Splunk
echo Updating Splunk to receive management data
"%PROGRAMFILES%\Splunk\bin\splunk.exe" set deploy-poll %DNSLOCATION%:8089 -auth admin:changeme

REM #########################################################################################################################################
REM ## Making a backup copy of the original trial license and then copying the forwarding license to the file Splunk looks at for licenses ##
REM #########################################################################################################################################
echo.
echo Setting up the license
rename "%PROGRAMFILES%\Splunk\etc\splunk.license" "splunk.license.bak"
copy "%PROGRAMFILES%\Splunk\etc\splunk-forwarder.license" "%PROGRAMFILES%\Splunk\etc\splunk.license" /y

REM ###############################################
REM ## Stopping the web service from starting up ##
REM ###############################################
echo. 
echo Disabling the Splunk web service
"%PROGRAMFILES%\Splunk\bin\splunk.exe" disable webserver -auth admin:changeme

REM #################################################################
REM ## Changing the Splunkd service account to restart if it fails ##
REM #################################################################
echo.
echo Setting the Splunk service to restart the service if it goes down.
sc \\%COMPUTERNAME% failure Splunkd actions= restart/300000/restart/300000 reset= 86400

REM ##############################################################
REM ## %PROGRAMFILES%\Splunk\etc\apps\sample_app\local\app.conf ##
REM ##############################################################
IF NOT EXIST "%PROGRAMFILES%\Splunk\etc\apps\sample_app\local\" mkdir "%PROGRAMFILES%\Splunk\etc\apps\sample_app\local\" 
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\sample_app\local\app.conf"
echo [install] >> "%PROGRAMFILES%\Splunk\etc\apps\sample_app\local\app.conf"
echo state = disabled >> "%PROGRAMFILES%\Splunk\etc\apps\sample_app\local\app.conf"

REM #############################################################
REM ## %PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf ##
REM #############################################################
IF NOT EXIST "%PROGRAMFILES%\Splunk\etc\apps\search\local\" mkdir "%PROGRAMFILES%\Splunk\etc\apps\search\local\"
echo.>> "%PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf"
echo [script://$SPLUNK_HOME\bin\scripts\splunk-wmi.py] >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf"
echo disabled = true  >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf"
echo.  >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf"
echo [script://$SPLUNK_HOME\bin\scripts\splunk-regmon.py] >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf"
echo disabled = true >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\inputs.conf"

REM ####################################################################
REM ## %PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf ##
REM ####################################################################
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf"
echo [Top five sourcetypes] >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf"
echo enableSched = false >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf"
echo [Indexing workload] >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf"
echo enableSched = false >> "%PROGRAMFILES%\Splunk\etc\apps\search\local\savedsearches.conf"

REM ##############################################################
REM ## %PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf ##
REM ##############################################################
IF NOT EXIST "%PROGRAMFILES%\Splunk\etc\apps\windows\local\" mkdir "%PROGRAMFILES%\Splunk\etc\apps\windows\local\"
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo [WinEventLog:Application] >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo disabled = true >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo.  >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo [WinEventLog:Security] >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo disabled = false >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo [WinEventLog:System] >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo disabled = false >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo [script://$SPLUNK_HOME\bin\scripts\splunk-admon.py] >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo disabled = true >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo [monitor://$WINDIR\WindowsUpdate.log] >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"
echo disabled = false >> "%PROGRAMFILES%\Splunk\etc\apps\windows\local\inputs.conf"

REM #######################################################################
REM ## Checking to see if this server requires File Integrity Monitoring ##
REM #######################################################################
cls
echo.
choice /C YN /T 15 /D N /N /M "Does this server require File Integrity Monitoring? [Y|(N)]: "

IF %ERRORLEVEL% == 1 goto FIMBASE
IF %ERRORLEVEL% == 2 goto BU

:FIMBASE
REM ##########################################################
REM ## c:\program files\Splunk\etc\system\local\inputs.conf ##
REM ## [fschange:<path to folder/file>]                     ##
REM ## recurse=true                                         ##
REM ## pollPeriod=<time in seconds - 1 to 3600>             ##
REM ##########################################################

IF NOT EXIST "%PROGRAMFILES%\Splunk\etc\system\local\" mkdir "%PROGRAMFILES%\Splunk\etc\system\local\"
echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo [fschange:%WINDIR%] >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo recurse=false >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo pollPeriod=3600 >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo [fschange:%WINDIR%\system32] >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo recurse=true >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo pollPeriod=3600 >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"

choice /C YN /T 15 /D N /N /M "Do you require more than the default folders? [Y|(N)]: "
IF %ERRORLEVEL% == 1 goto FIM
IF %ERRORLEVEL% == 2 goto BU

REM ##########################################
REM ## Setting up File Integrity Monitoring ##
REM ##########################################
:FIM
echo Please list the first folder to be monitored:
set /p FIMFOLDER=
echo.
choice /C YN /T 15 /D N /N /M "Recrusive from this folder down? [Y|(N)]: "
IF %ERRORLEVEL% == 1 set FIMRECRUSIVE=true
IF %ERRORLEVEL% == 2 set FIMRECRUSIVE=false
echo.
echo How often, in seconds, will the files and folder need to be checked?
set /p FIMPOLLING=

echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo [fschange:%FIMFOLDER%] >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo recurse=%FIMRECRUSIVE% >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo pollPeriod=%FIMPOLLING% >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"

choice /C YN /T 15 /D N /N /M "Do you have other folders to monitor? [Y|(N)]: "
IF %ERRORLEVEL% == 1 goto FIM
IF %ERRORLEVEL% == 2 goto BU

:BU
REM ################################################################
REM ## Setting the index to go to the correct BU - asking what BU ##
REM ################################################################

echo.
echo Please select the BU for this server:
echo   1. Sales
echo   2. Marketing
echo   3. Domain Controllers
echo   4. Web Servers
echo   5. Linux
echo   6. Windows

choice /c:123456

IF %ERRORLEVEL% == 6 set BU=windows
IF %ERRORLEVEL% == 5 set BU=linux
IF %ERRORLEVEL% == 4 set BU=webserver
IF %ERRORLEVEL% == 3 set BU=domaincontrollers
IF %ERRORLEVEL% == 2 set BU=marketing
IF %ERRORLEVEL% == 1 set BU=sales

REM ####################################################################
REM ## Setting the index to go to the correct BU - Creating the index ##
REM ####################################################################

"%PROGRAMFILES%\Splunk\bin\splunk.exe" add index %BU% -auth admin:changeme

REM ###################################################################
REM ## Setting the index to go to the correct BU - Setting the files ##
REM ## "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"          ##
REM ###################################################################

echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo [default] >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"
echo index = %BU% >> "%PROGRAMFILES%\Splunk\etc\system\local\inputs.conf"

REM ##########################################
REM ## Setting the system to only use SSLv3 ##
REM ##########################################

echo. >> "%PROGRAMFILES%\Splunk\etc\system\local\server.conf
echo supportSSLV3Only = true >> "%PROGRAMFILES%\Splunk\etc\system\local\server.conf"

goto RESTART

REM ####################
REM ## Restart Splunk ##
REM ####################

:RESTART

REM ########################################
REM ## Setting the new password in Splunk ##
REM ########################################
echo.
echo Setting up the admin account
"%PROGRAMFILES%\Splunk\bin\splunk.exe" edit user admin -password TheNewPassword -auth admin:changeme -roles admin

REM #########################
REM ## Stopping Splunk Web ##
REM #########################
cls
echo.
echo Stopping the Splunk web interface
net stop Splunkweb

echo.
echo Restarting Splunk
"%PROGRAMFILES%\Splunk\bin\splunk.exe" restart
%WINDIR%\temp\portqry -n %COMPUTERNAME% -e 8089 -q
IF %ERRORLEVEL% == 0 goto COMPLETE
net start splunkd
%WINDIR%\temp\portqry -n %COMPUTERNAME% -e 8089 -q
IF %ERRORLEVEL% == 0 goto COMPLETE
goto COMPLETEWITHERRORS

REM ####################
REM ## Missing Files  ##
REM ####################
:MISSINGFILES
cls
Echo Looks like the file(s) required are not in the same place as this script.
echo Please make sure the MSI installer file is in the same place as this script.
echo Also, make sure you have put all the files in the %WINDIR%\temp\ directory 
echo and that you have ran the install command file from the %WINDIR%\temp\ directory.
echo.
echo Required File:
echo PortQry.exe
echo.
echo and one of these files:
echo %Splunk32%
echo %Splunk64%
goto END

REM #############################
REM ## Splunk is now installed ##
REM #############################
:COMPLETE
cls
echo Splunk is installed.
goto END

:COMPLETEWITHERRORS
cls
echo Splunk is installed but was not able to start.  Please check why it didn't start.
goto END

REM ####################
REM ## END of program ##
REM ####################
:END
echo.
pause

[thall79]

From the server command prompt I run:

msiexec.exe /i "where the install files are located" INSTALLDIR="where you want splunk installed" FORWARD_SERVER="IP or DNS:Port_used" SPLUNK_APP="SplunkLightForwarder" /passive

This is how I have been setting up my SplunkForwarder for windows. Then after I copy my custom .conf and disable Splunk Web from services.msc and a splunkd restart I am up and running.

Travis.

[simonmag]

I was looking at impementing Splunk a couple of months ago and I thought about how I was going to get the Security log from over 200 servers. I to thought about installing forwarders on each of the servers but this would have been impossible to do in the time that I had.

I ended up using WMI to gather selected events from Security logs, I had to build three Splunk servers - 1 central server that would do all the indexing, 1 that would gather from non-domain servers and one that gathers from AD servers each of these forward to the central instance for indexing, We are also indexing all the syslog data on UDP:514.

[matt_1]

For various reasons we don't have a viable software distribution system. To get around that we leveraged a few open source tools and wrote a few cmd scripts.

[Tools] blat.exe psexec.exe

[Overall Iterative Install/Uninstall Wrapper]

echo off
set runPath=%~dp0
set hostsFile=targetHosts.txt
set adminName=%1
set adminPass=%2
set action=%3
set cmdExe=c:\winnt\system32\cmd.exe /C
set installerPath=\\<host>\<share>$\<subPath>
set installCmd=%cmdExe% %installerPath%\splunkForwarderInstall.cmd
set uninstallCmd=%cmdExe% %installerPath%\splunkForwarderUninstall.cmd

if "%adminName%"=="" GOTO USAGE
if "%adminPass%"=="" GOTO USAGE
if "%action%"=="install" GOTO INSTALL
if "%action%"=="uninstall" GOTO UNINSTALL
GOTO USAGE

:INSTALL
echo Installing Splunk Forwarders:
echo CMD: %runPath%psexec @%runPath%%hostsFile% -u %adminName% -p %adminPass% %installCmd%
%runPath%psexec @%runPath%%hostsFile% -u %adminName% -p %adminPass% %installCmd%
GOTO EOF

:UNINSTALL
echo Uninstalling Splunk Forwarders:
echo CMD: %runPath%psexec @%runPath%%hostsFile% -u %adminName% -p %adminPass% %uninstallCmd%"
%runPath%psexec @%runPath%%hostsFile% -u %adminName% -p %adminPass% %uninstallCmd%"
GOTO EOF

:USAGE
echo "%0 <adminName> <adminPass> <install|uninstall>"

:EOF
pause

[Installer CMD Script]

@echo off

rem Identify execution path
set runPath=%~dp0

rem Define installation constants
set installPath=<Target Installation Path>
set installPkg=splunk-<version>-release.msi

rem Define SMTP messaging constants.  For multiple message recipients,
rem define a comma (,) seperated list of SMTP addresses in %smtpRecip%
set smtpSrvr=<SMTP Gateway Server>
set smtpPort=<SMTP Server Port>
set smtpRecip=<SMTP Recipient Addresses>
set smtpSub=Splunk Forwarder Installation

set smtpBody=%installPath%\..\smtpBody.txt
set installer=%runPath%%installPkg%

echo =============================================== >> %smtpBody%
echo [+] Grab simple host and user information > %smtpBody%
mkdir %installPath% >> %smtpBody%
mkdir %installPath%\etc\apps\DeployClientInfo >> %smtpBody%
echo %USERNAME% >> %smtpBody%
echo %COMPUTERNAME%.%USERDNSDOMAIN% >> %smtpBody%
ipconfig >> %smtpBody%
echo =============================================== >> %smtpBody%
set msiCmd=msiexec.exe /log %installPath%\SplunkInst.log /i "%installer%" INSTALLDIR="%installPath%" SPLUNK_APP="" /passive
echo [+] Install Splunk Forwarder >> %smtpBody%
echo : %msiCmd% >> %smtpBody%
msiexec.exe /log %installPath%\SplunkInst.log /i "%installer%" INSTALLDIR="%installPath%" SPLUNK_APP="" /passive >> %smtpBody%
copy %installPath%\etc\splunk-forwarder.license %installPath%\etc\splunk.license >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Shutdown Splunk services and disable Splunk Web >> %smtpBody%
sc stop Splunkweb >> %smtpBody%
sc config Splunkweb start= disabled >> %smtpBody%
sc stop Splunkd >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Copy confiturations over those deployed by the msi installer >> %smtpBody%
xcopy %runPath%Splunk %installPath% /Y /R /H /S >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Start up the Splunkd service >> %smtpBody%
sc start Splunkd >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Complete... >> %smtpBody%
echo [+] See "%installPath%\SplunkInst.log" for installation details >> %smtpBody%
echo [+] Notifying the Splunk administrators about the installation >> %smtpBody%
"%runPath%blat.exe" "%smtpBody%" -server "%smtpSrvr%" -port "%smtpPort%" -f "%USERNAME%@%COMPUTERNAME%.%USERDNSDOMAIN%" -t "%smtpRecip%" -s "%smtpSub%" -attach "%installPath%\SplunkInst.log"

rem Cleanup trace file
del %smtpBody%

[Uninstaller CMD Script]

@echo off

rem Identify execution path
set runPath=%~dp0

rem Define installation constants
set installPath=<Target Installation Path>
set installPkg=splunk-<version>-release.msi

rem Define SMTP messaging constants.  For multiple message recipients,
rem define a comma (,) seperated list of SMTP addresses in %smtpRecip%
set smtpSrvr=<SMTP Gateway Server>
set smtpPort=<SMTP Server Port>
set smtpRecip=<SMTP Recipient Addresses>
set smtpSub=Splunk Forwarder Installation

set smtpBody=%installPath%\..\smtpBody.txt
set installer=%runPath%%installPkg%

echo =============================================== >> %smtpBody%
echo [+] Grab simple host and user information > %smtpBody%
echo %USERNAME% >> %smtpBody%
echo %COMPUTERNAME%.%USERDNSDOMAIN% >> %smtpBody%
ipconfig >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Shutdown Splunk services and disable Splunk Web >> %smtpBody%
sc stop Splunkweb >> %smtpBody%
sc stop Splunkd >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Uninstall Splunk Forwarder >> %smtpBody%
echo : msiexec.exe /x "%installer%" /passive >> %smtpBody%
msiexec.exe /x "%installer%" /passive >> %smtpBody%
echo =============================================== >> %smtpBody%
echo [+] Complete... >> %smtpBody%
echo [+] See "%installPath%\SplunkInst.log" for installation details >> %smtpBody%
echo [+] Notifying the Splunk administrators about the installation >> %smtpBody%
"%runPath%blat.exe" "%smtpBody%" -server "%smtpSrvr%" -port "%smtpPort%" -f "%USERNAME%@%COMPUTERNAME%.%USERDNSDOMAIN%" -t "%smtpRecip%" -s "%smtpSub%"

rem Cleanup trace file
del %smtpBody%

It took me about 5-10 minutes to install the forwarders across 40 servers. As part of the installation package, we pre-created all of the splunkd configurations (deployment client, logging) which are copied as part of the installation. Our particular installation we made sure to disable SplunkWeb, and have splunkd check into our deployment server for all monitoring details. This includes the instructions as to which index server/s to connect to.

[gkanapathy]

I'm keeping this: http://www.splunk.com/wiki/Deploying_Splunk_Light_Forwarders

up to date as the recommended way to deploy. Please refer to that link, since as Splunk versions change and we develop new practices, I'll maintain the other link.

---

Note the the below applies to 4.0. For 4.1, the script is valid, but the configuration file set required is smaller, as 4.1 does not set as much upon install, and therefore does not need as much to disable functionality. In particular, you will not need:

etc\apps\search\local\inputs.conf
etc\apps\search\local\savedsearches.conf
etc\apps\SplunkLightForwarder\local\inputs.conf
etc\apps\windows\local\inputs.conf
etc\apps\windows\local\savedsearches.conf

but you will still want to have:

etc\apps\sample_app\local\app.conf
etc\apps\SplunkLightForwarder\local\app.conf
etc\apps\deploymentclient\local\deploymentclient.conf

---

You can use any software distribution mechanism you have available. To install Light Forwarders on Windows, I recommending creating a Windows batch file install.cmd containing:

setlocal
IF "%PROCESSOR_ARCHITECTURE%" == "AMD64" goto b64
IF "%PROCESSOR_ARCHITEW6432%" == "AMD64" goto b64
:b32
set SPLUNK_MSI=%~dp0\splunk-4.0.9-74233-x86-release.msi
REM set above to path to 32-bit version
goto endb6432

:b64
set SPLUNK_MSI=%~dp0\splunk-4.0.9-74233-x64-release.msi
REM set above to path to 64-bit version
:endb6432

if not defined ProgramFilesW6432 (    
    set LOC=%ProgramFiles%\Splunk
) else (
    set LOC=%ProgramFilesW6432%\Splunk
)

rem the WINEVENLOG*CHECK settings don't do anything in 4.0. They're just carried over from 3.x where they make sure the inputs weren't enabled.
msiexec.exe /i "%SPLUNK_MSI%" INSTALLDIR="%LOC%" LAUNCHSPLUNK=0 WINEVENTLOGAPPCHECK=0 WINEVENTLOGSYSCHECK=0 WINEVENTLOGSECCHECK=0 SPLUNK_APP="" /QUIET 
xcopy "%~dp0\etc" "%LOC%\etc" /s /f /y
copy /y "%LOC%\etc\splunk-forwarder.license" "%LOC%\etc\splunk.license"
pushd "%LOC%\bin\"
splunk restart --accept-license --no-prompt --answer-yes

popd
endlocal

Edit the Splunk MSI path information as appropriate, and place the MSI files in the same directory as this script. Finally, create a directory in the same folder called etc, and place all your initial Splunk configuration files there.

To create a bare initial installation that does nothing, with all defaults inputs and scripts disabled, place the following files into the etc directory. Other configurations are possible with different sets of configuration files:

etc\apps\sample_app\local\app.conf
etc\apps\search\local\inputs.conf
etc\apps\search\local\savedsearches.conf
etc\apps\SplunkLightForwarder\local\app.conf
etc\apps\SplunkLightForwarder\local\inputs.conf
etc\apps\windows\local\inputs.conf
etc\apps\windows\local\savedsearches.conf
etc\apps\deploymentclient\local\deploymentclient.conf

The files should contain:

etc\apps\sample_app\local\app.conf:

[install]
state = disabled

etc\apps\search\local\inputs.conf:

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.py]
disabled = true

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.py]
disabled = true

etc\apps\search\local\savedsearches.conf:

[Top five sourcetypes]
enableSched = false

[Indexing workload]
enableSched = false

etc\apps\SplunkLightForwarder\local\app.conf:

[install]
state = enabled

etc\apps\SplunkLightForwarder\local\inputs.conf:

[monitor://$SPLUNK_HOME\var\log\splunk]
disabled = true

etc\apps\windows\local\inputs.conf:

[WinEventLog:Application]
disabled = true

[WinEventLog:Security]
disabled = true

[WinEventLog:System]
disabled = true

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.py]
disabled = true

[monitor://$WINDIR\WindowsUpdate.log]
disabled = true

etc\apps\windows\local\savedsearches.conf:

[CPU Utilization Summary]
enableSched = 0

[CPU Utilization by Threshold]
enableSched = 0

[Top Processes by Memory]
enableSched = 0

[Top Processes by CPU]
enableSched = 0

[Disk Utilization]
enableSched = 0

[win_eventlog_count_sum_index]
enableSched = 0

[performance_snapshot]
enableSched = 0

and finally, etc\apps\deploymentclient\local\deploymentclient.conf is only necessary if you will be using the Splunk Deployment Server to manage the forwarder configurations. Unless you have another configuration management system, we recommend you use the Deployment Server to be able to make modifications to forwarder configurations from. However, any system which lets you push configurations files and restart a service can be used instead, e.g., on Linux systems, puppet has been a good and effective solution that can both install Splunk and manage the configuration.

[deployment-client]

[target-broker:deploymentServer]
targetUri = mydeploymentserverhostname:8089

[balbano]

I don't think that is the issue... if I manually setup splunk (non-forwarder mode) and hardwire inputs.conf and outputs.conf in SplunkHome/etc/system/local it works fine and sends the logs to my indexers... that is not the case in light forwarder or even forwarder mode...

Don't see how it can be UAC related if the same local system user is being used by default. Should I open a support case for this?

Let me know.

Thanks.

Brian

[gkanapathy]

It is possible that Windows User Account Control is blocking this. The Splunk MSI in the current release I believe will simply fail silently if UAC fails. You may try checking into this.

[balbano]

I have gotten this to work for my Server 2003 boxes but not my Server 2008 R2 64 bit box. Has anyone else had the same issue???