Is it okay to run a universal forwarder without an inputs.conf?

[xavierashe]

I am the security guy and Splunk admin. I am running 6.6.x universal forwarders on all my windows servers. I just found out that the server admins are cloning boxes all willy-nilly. When trying to figure out why SERVER05 wasn't reporting in, it was because its inputs.conf had "host = SERVER01". I was getting my data, it was just hiding under the wrong host.

Googling around I found the solution is to delete inputs.conf and server.conf, then restart the UF. This seems to work. The UF does recreate the server.conf, but not the inputs.conf.

My question is, is this a problem? All of my inputs are managed in apps via a deployment server. Do I need an inputs.conf that specifies the hostname? I can't see any problems right now but wanted to ask the community.

[xavierashe · Accepted solution]

I wanted to come back to this post to let everyone know that my solution works fine except for Stream. I use Stream to collect DNS logs. When i deleted inputs.conf and server.conf, the host came in as literal string"$decideatstartup". All other event types works as advertised.

Here's the script I wrote to clean up our environment. I have it scheduled to run once a week.

$Date    = Get-Date -Format yyyyMMdd
$logFile = "c:\work\logs\cleanupSplunk-"+$Date+".log"

function logger{
    param ([string]$element, [string]$message)
    $string = "$(Get-Date); $($element); $($message)"
    Add-Content -path $logFile $string
}

function fix-Splunk{
    param( [string]$server )
    $clean = $true
    if(![System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\bin\splunkd.exe")){
        logger $server "WARNING Splunk not found"
    } else {
        $splunkService = Get-Service -Computer $server -Name splunkforwarder
        if([System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf")){
            $clean = $false
            logger $server "INFO inputs.conf found"
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR deleting inputs.conf"
            } else {
                logger $server "INFO inputs.conf deleted"
            }
            Clear-Variable ProcessError
            Restart-Service -InputObject $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR restarting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder restarted"
            }
            Clear-Variable ProcessError
        }
        if([System.IO.File]::Exists("\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\deploymentclient.conf")){
            $clean = $false
            logger $server "deplymentclient found"
            Remove-Item –path "\\$server\c$\Program Files\SplunkUniversalForwarder\etc\system\local\deploymentclient.conf" -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR deleting deplymentclient.conf"
            } else {
                logger $server "INFO deplymentclient.conf deleted"
            }
            Clear-Variable ProcessError
            Restart-Service -InputObject $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR restarting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder restarted"
            }
            Clear-Variable ProcessError
        }
        if ($splunkService.Status -eq "Stopped") {
            $clean = $false
            logger $server "WARNING Splunk service stopped"
            Start-Service $splunkService -ErrorAction SilentlyContinue -ErrorVariable ProcessError
            if ($ProcessError) {
                logger $server "ERROR starting splunkforwarder"
            } else {
                logger $server "INFO splunkforwarder started"
            }
            Clear-Variable ProcessError
        }
        if ($clean -eq $true) {
            logger $server "INFO server clean"
        }
    }
}

logger "-" "INFO Script Started"
Get-ADComputer -ResultPageSize 10000 -ResultSetSize $null -Filter { 
    (OperatingSystem -Like '*Windows*Server*') -and
    (enabled -eq $True)
    } -ErrorAction SilentlyContinue | where { ($_.DistinguishedName -notlike "*OU=Domain Controllers,*") } | select-object -expandproperty name | foreach-object {
    Write-Warning "Trying $_"
    if (test-connection -computername $_ -count 1 -quiet -ErrorAction SilentlyContinue) {
        fix-Splunk $_
    } else {
        logger $_ "ERROR cannot connect"
    }
}
logger "-" "INFO Script Finished"

[ddrillic]

-- Do I need an inputs.conf that specifies the hostname?

You don't need to explicitly specify the hostname - the forwarder runs uname -a on Linux and a similar command on Windows to find out, by itself, the host name.

[s2_splunk]

You should work with your server admins and have them run ./splunk clone-prep-clear-config if they clone an existing forwarder installation, then restart it (Documentation). That will not only initialize host settings, but also generate a new GUID for the installation.

[xavierashe]

Yes, absolutely. That's the plan moving forward. But I've got fix a few hundred machines in the mean time.

[skoelpin]

You can easily create a new inputs.conf under /opt/splunk/etc/system/local

So essentially you will have multiple inputs.conf.. The one under etc/system/local just defines the hostname so you can identify it

Here's what it should look like

[default]
host = <HOST-NAME>

Restart splunkd after making this change

[xavierashe]

True, but I've got to fix a few hundred machines. I need something scriptable. While I could do what you suggest, the real question is "do I need to."