Hi forum, I have a 2 peer single site (sf2, rf2) index cluster. We recognized that the primaries for indexes are not distributed even by using the search:     | rest splunk_server=local /services/cluster/master/buckets | rex field=title "^(?<repl_index>[^\~]+)" | search repl_index="*" standalone=0 frozen=* | rename title AS bucketID | fields bucketID peers.*.search_state peers.*.bucket_flags frozen repl_index | rename peers.3DAB62DE-6D21-4C93-B8E5-A65370709B79.bucket_flags as bucketflags | eval prim=if(bucketflags = "0x0","prim_yes","prim_no") | stats count by repl_index prim | xyseries repl_index prim count | fillnull prim_yes,prim_no | eval ratio=prim_yes/(prim_yes+prim_no) | eval ratio=round(ratio*100,2) | search repl_index="*"       More or less all primaries are either on one indexer or the other, resulting in uneven load as we have a search hotspot on one index. We were able to have a far better distribution after we set sf=1, removed excess buckets and set sf=2 again. Unfortunatly after stop an indexer for a while or do a rolling restart it's again very uneven distributed (as seen on the first screenshot). it's also possible to get an even distribution when stopping clustermaster and peers at the same time and starting again - in this time we have data loss. restarting any component for it's own doesn't fix the issue. we tried to rebalance primaries using:     curl -k -u admin:plaseentercreditcardnumber --request POST https://localhost:8089/services/cluster/master/control/control/rebalance_primaries     any hints how to fix this? We are using v8.0.7. best regards, Andreas ... View more

Hi all, I'm having non-indexed-extracted json in events. When there is a json "host" field host, which is different from the indexed "host", then the search view is showing you 2 values for host in smart or verbose mode. you can't work with the searchtime extracted json host field - clicking on it gives you no results - as host is an indexed field.  .. when you are doing a ... | stats count by host, then only "indextimehost" is reported back - as expected.   this behaviour differers from "normal" kv searchtime detection:   i found multiple posts regarding this like: https://community.splunk.com/t5/Getting-Data-In/Duplicate-host-field-after-indexing-JSON-event/m-p/292472 unfortunately i'm not able to change the json field name at the source. Rewriting is also no good option for me.  This more looks like a display bug for me.. but drives the poweruser crasy.   best Regards, Andreas ... View more

Hi all, I'm receiving a lot of splunk-optimize errors in splunkd.log.  ls -l /opt/splunk/var/lib/splunk/audit/db/hot_v1_455 | grep tsidx | wc -l 105 [root@indexer-2 splunk]# /opt/splunk/bin/splunk-optimize -v -d /opt/splunk/var/lib/splunk/audit/db/hot_v1_455 Logging configuration: verbose=1, log2splunk=0 tm= 1610364481 INFO splunk-optimize start: dir=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455 mode=0 isfinal=false max_iteration=2147483647 min_src_count=8 lex_tpb=64 write_level=1 target_size=1572864000 tm= 1610364481 DEBUG source_0=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610362948-1610362948-15554879922328683899.tsidx sz=1080 tm= 1610364481 DEBUG source_1=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363147-1610363147-16387064457660043774.tsidx sz=1168 tm= 1610364481 DEBUG source_2=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363267-1610363267-16901658911893982757.tsidx sz=1168 tm= 1610364481 DEBUG source_3=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363272-1610363272-16923132382574368602.tsidx sz=1176 tm= 1610364481 DEBUG source_4=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363092-1610363092-16150633440092436534.tsidx sz=1176 tm= 1610364481 DEBUG source_5=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363178-1610363178-16519774093793408615.tsidx sz=1176 tm= 1610364481 DEBUG source_6=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363057-1610363057-16000335144082864920.tsidx sz=1176 tm= 1610364481 DEBUG source_7=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363122-1610363122-16279466559003501729.tsidx sz=1176 tm= 1610364481 DEBUG source_8=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363058-1610363058-16004583287645382032.tsidx sz=1176 tm= 1610364481 DEBUG source_9=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363212-1610363212-16665396587756881875.tsidx sz=1176 tm= 1610364481 DEBUG source_10=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363152-1610363152-16408391753423458229.tsidx sz=1176 tm= 1610364481 DEBUG source_11=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363242-1610363242-16797814191883895619.tsidx sz=1176 tm= 1610364481 DEBUG source_12=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363238-1610363238-16777050395829684027.tsidx sz=1176 tm= 1610364481 DEBUG source_13=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363062-1610363062-16021755056521050397.tsidx sz=1176 tm= 1610364481 DEBUG source_14=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363298-1610363298-17035162030350365663.tsidx sz=1176 tm= 1610364481 DEBUG source_15=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363182-1610363182-16537360674905484228.tsidx sz=1176 tm= 1610364481 DEBUG source_16=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363288-1610363288-16996035449514477598.tsidx sz=1224 tm= 1610364481 DEBUG source_17=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363221-1610363221-16704811249229177472.tsidx sz=1240 tm= 1610364481 DEBUG source_18=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363248-1610363248-16826109427840005476.tsidx sz=1248 tm= 1610364481 DEBUG source_19=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363226-1610363226-16726339824340034748.tsidx sz=1248 tm= 1610364481 DEBUG source_20=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363103-1610363103-16197833266659683219.tsidx sz=1248 tm= 1610364481 DEBUG source_21=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363038-1610363038-15927433915388732711.tsidx sz=1248 tm= 1610364481 DEBUG source_22=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363278-1610363278-16953970350838872388.tsidx sz=1248 tm= 1610364481 DEBUG source_23=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363048-1610363048-15965922895444683963.tsidx sz=1248 tm= 1610364481 DEBUG source_24=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363296-1610363296-17026567070646661147.tsidx sz=1248 tm= 1610364481 DEBUG source_25=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363268-1610363268-16910815734924420355.tsidx sz=1248 tm= 1610364481 DEBUG source_26=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363218-1610363218-16695936197528204421.tsidx sz=1248 tm= 1610364481 DEBUG source_27=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363118-1610363118-16266549629544376987.tsidx sz=1248 tm= 1610364481 DEBUG source_28=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363138-1610363138-16352697168739542857.tsidx sz=1248 tm= 1610364481 DEBUG source_29=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363084-1610363084-16116233719873926123.tsidx sz=1248 tm= 1610364481 DEBUG source_30=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363238-1610363238-16782065727925460598.tsidx sz=1248 tm= 1610364481 DEBUG source_31=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363174-1610363174-16502826878692865729.tsidx sz=1248 tm= 1610364481 ERROR optimize finished: failed, see rc for more details, dir=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455, rc=-29 (unsigned 227), errno=2 tm= 1610364481 INFO exiting splunk-optimize process with rc=-29 (unsigned 227)   I haven't found any documentation what returncode -29 means or how to get more logs for this issue. Could anybody help out? Regards, Andreas ... View more