Oh my, you are right, I made a stupid mistake! I so used to use props.conf in the most (personal) cases that I automatically tried to use sourcetype in stanza name. But I need the input of splunk UF introspection, that generates splunkd 'sourcetype'. And this monitor named 'monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log' ... View more

Still feel myself as a dummy I set this in the UF inputs.conf: [splunkd] _meta = my_field::abc   Then I set this in the feilds.conf of SH: [my_field] INDEXED = true INDEXED_VALUE = false   Of course, I applied config on both instances and still can't find new field created in "_meta" ... View more

Thank you for such a detailed explanation. But let me ask a little bit more... Did I correctly understand that under "source forwarder" you meant UF? I am asking because I tried to set meta = testfield::test_value in two different places, in the inputs.conf of the SplunkUniversalForwarder app and in the system inputs.conf, and I couldn't find this new field during the search. ... View more

Thank you for the idea; this is a pretty easy solution. Obviously less complicated than I came up with =D ... View more

@isoutamo, Thank you for your attention to my problem. I saw this post, and I also saw the resolution—create the user 'system'. But my case is a little bit different because errors have no information about the user that is absent. Only quotes without anything. ... View more

Sorry for had being annoying, I'm stopping this behavior. ... View more

UP ... View more

Thank you for your thoughts, colleagues. I will check the idea of @dural_yyz that mentioned the absence of an EOF marker. @PickleRick , talking about permission, I'm pretty sure that this is not the case because about a month ago I found out that new Splunk UFs started to use "USE_LOCAL_SYSTEM = 0" by default during silent install. Because of it I was observing something like this on the affected UF instances: 10-27-2024 21:50:16.756 +0300 ERROR TailReader [3644 tailreader0] - Unable to remove sinkhole file: path=E:\path\file.xml, errno=Access is denied.   ... View more

Up A week ago, I tried to enable DEBUG log to find the root case But found only the similar events without anything helpful to find the root case ... View more

Thanks for the valuable advertising! I don't need this app in general, but I can use queries from dashboard as examples to resolve my particular case. ... View more

Up ... View more

Up ... View more

Just a day ago, I migrated from 9.1.5 to 9.1.6. I confirm an absence of zombie processes! 😃 ... View more

I think that your are mistaken According to the below message of weiss_h, this issue fixed only in new version 9.3.1 ... View more

I'm familiar with the settings above. I use no_priority_stripping on all udp inputs because I have some logic with PRI codes processing. ... View more

One additional question. Do you know any free products that can act like a syslog server (listen to udp\tcp and store files) but working on Windows? ... View more

I'm really not an expert in datagrams, but according my observation behavior, it is not true If each datagram is separate event, it's not possible to see the same behavior because with "SHOULD_LINMERGE = false" events can be defined without LINE_BREAKER >each datagram is treated as separate event Returning to the _indextime versus _teme - it's just an addition in my case, because if your log rate is pretty low, you can see in realtime how events showing in search only after the next one ... View more

Thank you for your response. I understand that using a dedicated syslog server is the best practice, but until this moment I haven't understood with which errors I can come across without it. I tested your props.conf suggestion but still observe the same behavior that was described in the OP.   ... View more

Thank you for sharing it! I wrote a little script with simple logic using this workaround. I have rolled it out across my environment and will monitor how it works. But I think that this approach is quite a safe way to "remediate" the problem before the Splunk team fixes it. function Stop-SplZombieProcess { [CmdletBinding()] param ( [Parameter(Mandatory = $true)] [string]$HostName, [Parameter(Mandatory = $false)] [int]$Threshold = 5000, [Parameter(Mandatory = $false)] [switch]$MultiZombie ) begin { } process { Write-Host "Trying to find zombies on the host '$HostName'." $Procs = Invoke-Command -ComputerName $HostName -ScriptBlock {Get-Process | Where-Object {$_.ProcessName -eq 'splunkd'} } if ($Procs.Count -eq 1) { Write-Host "Only one splunkd process with '$($Procs.Handles)' handles was found. Most likely it is not a zombie." } else{ [array]$Zombies = $Procs | Where-Object { $_.Handles -ge $Threshold } if ($Zombies) { if ($Zombies.Count -eq 1) { $ProcId = $Zombies.Id Write-Host "Zombie was found. The number of handles is '$($Zombies.Handles)'. Trying to kill." Invoke-Command -ComputerName $HostName -ScriptBlock {Stop-Process -Id $using:ProcId -Force} Write-Host "The zombie process with ProcId '$ProcId' has been killed on the host '$HostName'." } elseif ($MultiZombie) { Write-Host 'Performing zombie multikill.' foreach ($Item in $Zombies) { $ProcId = $Item.Id Write-Host "Zombie was found. The number of handles is '$($Item.Handles)'. Trying to kill." Invoke-Command -ComputerName $HostName -ScriptBlock {Stop-Process -Id $using:ProcId -Force} Write-Host "The zombie process with ProcId '$ProcId' has been killed on the host '$HostName'." } } else { Write-Warning "Found more than one process with handles more than '$Threshold'. Rise the threshold value or use the 'MultiZombie' switch to kill more than one zombie." } } else { Write-Host "Zombies not found on the host '$HostName'." } } } end { [System.gc]::collect() } } ... View more

Thank you for workaroud. I also applied it to one of my SHC member to see the difference. In the afternoon, a will check the existence of zombie procecess. ... View more