All Apps and Add-ons

Why is Splunk DB Connect 2 not rotating py_health.log on a Windows Heavy forwarder

ragedsparrow
Contributor

Splunk Version: 6.3.3
OS Version: Windows Server 2008 R2 Enterprise 64-Bit
DB Connect Version: 2.1.3

It seems when the py_health.log file reaches 10,000KB, it is not rotating out and seems to lock. This is causing issues with DB Connect connecting to the databases as well as running the queries. Has anyone else had this problem? I can stop Splunk and roll it manually, however it isn't ideal. Has anyone else had this issue?

1 Solution

ragedsparrow
Contributor

So, it seems that since this is a Windows OS, the file, when in use can not be modified or deleted. My workaround was to create a scheduled task to run a rotate script that will stop Splunk, remove the backup, rename the file, and start Splunk. This is probably not the best way to do it, but it's the only way that I've found so far.

$splunkProcess = Get-Service | Where-Object {$_.Name -eq "Splunkd"} | Where-Object {$_.Status -eq "Running"}

if ($splunkProcess){

Write-Host "Splunk is running..beginning process"
$StopCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' stop
'@
Invoke-Expression -Command:$StopCommand

    if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
        Write-Host "Splunk stopped, deleting old log file"
        Remove-Item -Path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
    }
    else{Write-Host "No file backup found"}
    Write-Host "Renaming log file"
    Rename-Item 'C:\Program Files\Splunk\var\log\splunk\py_health.log' 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
$StartCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Write-Host "Starting Splunk"
Invoke-Expression -Command:$StartCommand

}

else {
if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
Write-Host "Splunk is not running"
$command = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Invoke-Expression -Command:$command

}

This was pretty much just banged together to hold things together for now. I will be adding in some exception handling as well as removing the Write-host and will be instead writing to a log file.

View solution in original post

0 Karma

ragedsparrow
Contributor

So, it seems that since this is a Windows OS, the file, when in use can not be modified or deleted. My workaround was to create a scheduled task to run a rotate script that will stop Splunk, remove the backup, rename the file, and start Splunk. This is probably not the best way to do it, but it's the only way that I've found so far.

$splunkProcess = Get-Service | Where-Object {$_.Name -eq "Splunkd"} | Where-Object {$_.Status -eq "Running"}

if ($splunkProcess){

Write-Host "Splunk is running..beginning process"
$StopCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' stop
'@
Invoke-Expression -Command:$StopCommand

    if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
        Write-Host "Splunk stopped, deleting old log file"
        Remove-Item -Path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
    }
    else{Write-Host "No file backup found"}
    Write-Host "Renaming log file"
    Rename-Item 'C:\Program Files\Splunk\var\log\splunk\py_health.log' 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
$StartCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Write-Host "Starting Splunk"
Invoke-Expression -Command:$StartCommand

}

else {
if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
Write-Host "Splunk is not running"
$command = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Invoke-Expression -Command:$command

}

This was pretty much just banged together to hold things together for now. I will be adding in some exception handling as well as removing the Write-host and will be instead writing to a log file.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

This was just fixed in version 2.3.0.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...