Thank you for the idea; this is a pretty easy solution. Obviously less complicated than I came up with =D ... View more

@isoutamo, Thank you for your attention to my problem. I saw this post, and I also saw the resolution—create the user 'system'. But my case is a little bit different because errors have no information about the user that is absent. Only quotes without anything. ... View more

Sorry for had being annoying, I'm stopping this behavior. ... View more

UP ... View more

Thank you for your thoughts, colleagues. I will check the idea of @dural_yyz that mentioned the absence of an EOF marker. @PickleRick , talking about permission, I'm pretty sure that this is not the case because about a month ago I found out that new Splunk UFs started to use "USE_LOCAL_SYSTEM = 0" by default during silent install. Because of it I was observing something like this on the affected UF instances: 10-27-2024 21:50:16.756 +0300 ERROR TailReader [3644 tailreader0] - Unable to remove sinkhole file: path=E:\path\file.xml, errno=Access is denied.   ... View more

Up A week ago, I tried to enable DEBUG log to find the root case But found only the similar events without anything helpful to find the root case ... View more

Thanks for the valuable advertising! I don't need this app in general, but I can use queries from dashboard as examples to resolve my particular case. ... View more

Up ... View more

Up ... View more

Just a day ago, I migrated from 9.1.5 to 9.1.6. I confirm an absence of zombie processes! 😃 ... View more

I think that your are mistaken According to the below message of weiss_h, this issue fixed only in new version 9.3.1 ... View more

I'm familiar with the settings above. I use no_priority_stripping on all udp inputs because I have some logic with PRI codes processing. ... View more

One additional question. Do you know any free products that can act like a syslog server (listen to udp\tcp and store files) but working on Windows? ... View more

I'm really not an expert in datagrams, but according my observation behavior, it is not true If each datagram is separate event, it's not possible to see the same behavior because with "SHOULD_LINMERGE = false" events can be defined without LINE_BREAKER >each datagram is treated as separate event Returning to the _indextime versus _teme - it's just an addition in my case, because if your log rate is pretty low, you can see in realtime how events showing in search only after the next one ... View more

Thank you for your response. I understand that using a dedicated syslog server is the best practice, but until this moment I haven't understood with which errors I can come across without it. I tested your props.conf suggestion but still observe the same behavior that was described in the OP.   ... View more

Thank you for sharing it! I wrote a little script with simple logic using this workaround. I have rolled it out across my environment and will monitor how it works. But I think that this approach is quite a safe way to "remediate" the problem before the Splunk team fixes it. function Stop-SplZombieProcess { [CmdletBinding()] param ( [Parameter(Mandatory = $true)] [string]$HostName, [Parameter(Mandatory = $false)] [int]$Threshold = 5000, [Parameter(Mandatory = $false)] [switch]$MultiZombie ) begin { } process { Write-Host "Trying to find zombies on the host '$HostName'." $Procs = Invoke-Command -ComputerName $HostName -ScriptBlock {Get-Process | Where-Object {$_.ProcessName -eq 'splunkd'} } if ($Procs.Count -eq 1) { Write-Host "Only one splunkd process with '$($Procs.Handles)' handles was found. Most likely it is not a zombie." } else{ [array]$Zombies = $Procs | Where-Object { $_.Handles -ge $Threshold } if ($Zombies) { if ($Zombies.Count -eq 1) { $ProcId = $Zombies.Id Write-Host "Zombie was found. The number of handles is '$($Zombies.Handles)'. Trying to kill." Invoke-Command -ComputerName $HostName -ScriptBlock {Stop-Process -Id $using:ProcId -Force} Write-Host "The zombie process with ProcId '$ProcId' has been killed on the host '$HostName'." } elseif ($MultiZombie) { Write-Host 'Performing zombie multikill.' foreach ($Item in $Zombies) { $ProcId = $Item.Id Write-Host "Zombie was found. The number of handles is '$($Item.Handles)'. Trying to kill." Invoke-Command -ComputerName $HostName -ScriptBlock {Stop-Process -Id $using:ProcId -Force} Write-Host "The zombie process with ProcId '$ProcId' has been killed on the host '$HostName'." } } else { Write-Warning "Found more than one process with handles more than '$Threshold'. Rise the threshold value or use the 'MultiZombie' switch to kill more than one zombie." } } else { Write-Host "Zombies not found on the host '$HostName'." } } } end { [System.gc]::collect() } } ... View more

Thank you for workaroud. I also applied it to one of my SHC member to see the difference. In the afternoon, a will check the existence of zombie procecess. ... View more

I also encountered exactly the same problem on my search head cluster. Now I'm on version 9.1.5 and still having this issue. ... View more

Thank you for your notable comment. I suspected that my configuration didn't work because of indexed extraction. But I hadn't time to check and I wasn't sure about it 😃 Talking about preamble, I tested settings that you mentioned a couple of times, but each time it worked worse than the nullQueue approach Maybe I just was not enough attentive...) ... View more

1. The first pair of props/transforms related to Universal Forwarder The second pair is putted on the indexer cluster layer 2. Yes, I see indexed extractions ... View more