I have cluster with 2 indexers, RF=2 running Splunk version 7.1.2 on Windows Server 2012. I often get following error: Indexer Clustering: too many bucket replication errors to target peer. In splunkd.log on both indexers I found similar errors: ERROR TimeInvertedIndex - Failed to rename from="C:\Splunk\var\lib\splunk\audit\db\hot_v1_278\.rawSize_tmp" to="C:\Splunk\var\lib\splunk\audit\db\hot_v1_278\.rawSize": Access is denied. ERROR LMApplyResponse - failed to rename C:\Splunk\var\lib\splunk\fishbucket\rawdata\1322324208-C:\Splunk\var\lib\splunk\fishbucket\rawdata\1322324208.old [1,1,1] (Access is denied.) ERROR HotBucketRoller - Unable to rename from='C:\Splunk\var\lib\splunk\_internaldb\db\hot_v1_572' to='C:\Splunk\var\lib\splunk\_internaldb\db\db_1570774736_1570503894_572_F0C749EE-B861-4598-B107-5358365E79D8' because The system cannot find the file specified. and so on. AND NO BUCKETS IN FIXUP STATE. Splunk is installed under local OS Admin (no AD). I checked file permissions for fishbucket\rawdata*.old files and found that no user or group has any access to it, even SYSTEM! My steps: 1. Removed fishbucket\rawdata\1322324208.old file 2. Executed icacls.exe commands in order to fix Splunk directory permissions: icacls.exe "C:\Splunk" /inheritance:e /T icacls.exe "C:\Splunk" /T /Q /reset 3. Initiated rolling restart on Master Node. Indexers entered maintenance mode and restarted. This caused a huge number of fixup tasks after restarts succeeded and fixed all issues for some time, but today I got same issues with other buckets. Why Splunk still creates files with bad access rights? What I can do in this situation? Maybe reinstall Splunk? Note: I know that running Splunk on Windows is pain and I can't move to Linux servers. I have 2 other clusters with indexers running on Windows and they do well. ... View more

I have splunk cluster with sf=2, rf=2 (they are met). It was maintained by another contractor, so I have no ideas about what caused the issue. First, on master node for some indexes I see data copies marked grey and data copies number is 170/173, etc... same for serachable and replicated copies. And no bucket fixup tasks are running. I guess this means that some buckets do not exist on both indexers, am I right? Second, I ran dbinspect command for this index, it returned 173 results. tsidxState is "full" for every bucket. How do I find problematic buckets and delete them to make Indexer clustering page green again? UPD. I uploaded photo ... View more

I have fully configured cluster running Splunk 6.6.5. All indexers and search heads work properly with other inputs. I added new TCP input where 5 workstations send their logs. But I got only one event indexed (from one ws, on one indexer) and new events never show up. I've tested network communication, all ports are open, telnet packets fly smoothly. I also checked if indexers' queues become blocked, but they are not. What could cause this issue? ... View more

I have some dashboard panels I want to export using their native "Export" button ( I don't speak about outputcsv command). Generated CSV file is always comma-separated. How can I set its default delimiter to ";"? I tried editing Splunk\Python-2.7\Lib\csv.py with no luck. ... View more

I wonder if I can use some fields instead of hard-typing category each time. I tried $result.category$, but it doesn't work. ... View more