All Apps and Add-ons

Why is Splunk DB Connect 2 not rotating py_health.log on a Windows Heavy forwarder

ragedsparrow
Contributor

Splunk Version: 6.3.3
OS Version: Windows Server 2008 R2 Enterprise 64-Bit
DB Connect Version: 2.1.3

It seems when the py_health.log file reaches 10,000KB, it is not rotating out and seems to lock. This is causing issues with DB Connect connecting to the databases as well as running the queries. Has anyone else had this problem? I can stop Splunk and roll it manually, however it isn't ideal. Has anyone else had this issue?

1 Solution

ragedsparrow
Contributor

So, it seems that since this is a Windows OS, the file, when in use can not be modified or deleted. My workaround was to create a scheduled task to run a rotate script that will stop Splunk, remove the backup, rename the file, and start Splunk. This is probably not the best way to do it, but it's the only way that I've found so far.

$splunkProcess = Get-Service | Where-Object {$_.Name -eq "Splunkd"} | Where-Object {$_.Status -eq "Running"}

if ($splunkProcess){

Write-Host "Splunk is running..beginning process"
$StopCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' stop
'@
Invoke-Expression -Command:$StopCommand

    if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
        Write-Host "Splunk stopped, deleting old log file"
        Remove-Item -Path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
    }
    else{Write-Host "No file backup found"}
    Write-Host "Renaming log file"
    Rename-Item 'C:\Program Files\Splunk\var\log\splunk\py_health.log' 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
$StartCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Write-Host "Starting Splunk"
Invoke-Expression -Command:$StartCommand

}

else {
if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
Write-Host "Splunk is not running"
$command = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Invoke-Expression -Command:$command

}

This was pretty much just banged together to hold things together for now. I will be adding in some exception handling as well as removing the Write-host and will be instead writing to a log file.

View solution in original post

0 Karma

ragedsparrow
Contributor

So, it seems that since this is a Windows OS, the file, when in use can not be modified or deleted. My workaround was to create a scheduled task to run a rotate script that will stop Splunk, remove the backup, rename the file, and start Splunk. This is probably not the best way to do it, but it's the only way that I've found so far.

$splunkProcess = Get-Service | Where-Object {$_.Name -eq "Splunkd"} | Where-Object {$_.Status -eq "Running"}

if ($splunkProcess){

Write-Host "Splunk is running..beginning process"
$StopCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' stop
'@
Invoke-Expression -Command:$StopCommand

    if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
        Write-Host "Splunk stopped, deleting old log file"
        Remove-Item -Path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
    }
    else{Write-Host "No file backup found"}
    Write-Host "Renaming log file"
    Rename-Item 'C:\Program Files\Splunk\var\log\splunk\py_health.log' 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'
$StartCommand = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Write-Host "Starting Splunk"
Invoke-Expression -Command:$StartCommand

}

else {
if (test-path 'C:\Program Files\Splunk\var\log\splunk\py_health.log.bak.log'){
Write-Host "Splunk is not running"
$command = @'
cmd.exe /c 'c:\Program Files\Splunk\bin\splunk.exe' start
'@
Invoke-Expression -Command:$command

}

This was pretty much just banged together to hold things together for now. I will be adding in some exception handling as well as removing the Write-host and will be instead writing to a log file.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

This was just fixed in version 2.3.0.

Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...