Ran into some issues upgrading our lab splunk environment and now I am concerned and hesitant to upgrade our prod cluster. In the past we have always performed upgrades in the following order: Deployment/license Server → Indexer Cluster Manager → Search Heads → Search Head Deployer → Indexers  → Heavy Forwarders In the past we only had one index cluster, but now we have two index clusters that our single search head cluster searches both against.  We were always able to do this upgrade order and have zero downtime in the past back when we had one index cluster, but it seems something about having a 2nd index cluster breaks this now once we start upgrading the searchheads.  To be clear, once we start the searchhead upgrade, we run splunk upgrade-init shcluster-members and also upgrade the searchheads one by one, making sure to put them in manual detention before hand and then bringing them back up afterward.   But this time, once we started upgrading the first searchhead, all searches broke in the UI giving this error: Unable to distribute to peer named <index_name_here> at uri=<ip_here>:8089 using the uri-scheme=https because peer has status=Down. To be clear - the state of the environment at this point at which the searches started failing, was: deployment server:  upgraded all index cluster managers:  upgraded searchhead deployer:  upgraded searchheads:  splunk upgrade-init shcluster-members initiated, and only 1 searchhead upgraded so far indexers:  Not upgraded heavy forwarders:  not upgraded Logs were still coming in, but we could no longer issue searches, always getting an error that a indexer was down  I checked the index cluster managers Index Clustering UI, and the indexers were showing as being UP, yet the indexes themselves were not displaying as being searchable.  I checked the cli with show cluster-status --verbose and all of the indexer was showing this weird state (For the "new" 2nd index cluster we only run 1 indexer in our lab environment): Pre-flight check successful .................. NO ├────── Replication factor met ............... YES ├────── Search factor met .................... YES ├────── All data is searchable ............... NO ├────── All peers are up ..................... YES ├────── CM version is compatible ............. YES ├────── No fixup tasks in progress ........... YES └────── Splunk version peer count { 10.0.2: 1 } └────── Ready for searchable rolling restart NO Indexing Ready YES HA Mode: Disabled <hostnamehere and guid> default Searchable YES Status Up Bucket Count=23 Splunk Version=10.0.2 _audit Number of non-site aware buckets=0 Number of buckets=9 Size=0 Searchable NO Replicated copies tracker 9/9 Searchable copies tracker 9/9 _internal Number of non-site aware buckets=0 Number of buckets=8 Size=0 Searchable NO Replicated copies tracker 8/8 Searchable copies tracker 8/8 _telemetry Number of non-site aware buckets=0 Number of buckets=2 Size=0 Searchable NO Replicated copies tracker 2/2 Searchable copies tracker 2/2 request Number of non-site aware buckets=0 Number of buckets=4 Size=0 Searchable NO Replicated copies tracker 4/4 Searchable copies tracker 4/4 I then checked logs on the index cluster manager itself, and noticed lines like this: WARN  GetBundleListTransaction Server <indexer_name>[https://<ip>:8089/services/admin/bundles/<clustermanager_name>] does not support bundle version listing. Probably an older version. Giving up due to error code 400. WARN  DistributedPeer Peer:https://<ip>:8089 Unable to get bundle list I thought ok, I guess maybe we'll need to upgrade the rest of the searchheads and it will clear the error then, and consider the indexes searchable?  Nope, powered through the rest of the searchheads and the error still persisted.  So then I upgraded the indexer in question for the 2nd cluster, and all it did was make it so the search ui then started throwing the same error for a random indexer in the "primary" cluster (which is 3 indexers for what it's worth).  And as I upgraded the individual indexer throwing the error it would just move on to the next one, so essentially that's why I say all the indexers basically had the issue, it just would throw the error on the first one it detected it on and stop moving through them. Once I upgraded all indexers service was restored and we could continue searching.  But this seems to direclty contradict what the splunk docs say, in that they say an index cluster manager should be able to be ahead of the indexers on the splunk version, and searchheads also should be able to be ahead of the indexers on splunk version.  This is why in the past we were able to upgrade it and have no downtime.  Is there something about having a searchhead search against two clusters that breaks this?  Or is there something about upgrading from 10.0.2 to 10.4.1 that breaks it?  For the record ingestion never broke during this time, only search. Can anyone help me out here?  Anything stand out, or has anyone had similar experiences after moving to a multi cluster environment? ... View more

I had to rebuild a searchhead some time back, everything works fine on the searchhead and it functions just fine within the searchhead cluster itself, however when I try to view the status of my searchheads via the Index Cluster Manager UI, this rebuilt searchhead always shows as "Down".  I am pretty certain it's because when I check the GUID locally on the searchhead listed here:   /opt/splunk/etc/instance.cfg It does not match the GUID the Index Cluster Manager has associated with that searchhead.  I cannot for the life of me figure out how to get the Cluster Manager to pick up the new guid though, does anyone know how I can accomplish this? ... View more

I have been tasked with building out new instances of anything that runs an older OS, and for our EC2 instances this most often means I have to rebuild the instance on Amazon Linux 2023.  What I've noticed, particularly with our heavy forwarders, is that they are constantly dying and ballooning memory to 100% with no rhyme or reason as to why (not heavy load, happens to random servers at random times, and there should be no world where the memory even comes close to getting consumed considering the low level of traffic for the servers in question which are in our lab environment).   We're running splunk version 10.0.2 Nothing helpful ever in the logs aside from indications that it ran OOM.  I've randomly tested with making absurdly large instances to see if the issue still occurs and sure enough it does, there's not enough memory in the world to handle whatever error seems to be occurring, so it seems to be something more akin to a bug that's occurring. I'm curious if anyone else has ran into issues with AL2023 and running Splunk? ... View more

Lately our searchheads will run into issues where the srtemp folder baloons to 80+GB and fills the local hard drive.  To temporarily fix, I have to shut down splunk and then run the splunk command: clean-srtemp  It will then be good for several days to a week and eventually it's back again, rinse and repeat.  I'm curious what my options are for determining what is causing the srtemp folder to fill consistently? ... View more

This is an odd one happening on each of our indexers.  The same behavior happens quite frequently, where we will get exactly 11 of these Remote token requests from splunk-system-user, and exactly 1 of them will fail.  Here is how it looks in the audit logs. 04-22-2024 21:30:31.964 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:31.964, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:31.986 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:31.986, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:32.384 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:32.384, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:32.395 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:32.395, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:40.687 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:40.687, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:40.694 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:40.694, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:46.803 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:46.803, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:46.815 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:46.815, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:47.526 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:47.526, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:47.542 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:47.542, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:55.317 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:55.317, user=splunk-system-user, action=Remote token requested, info=failed] My problem is I can't do much more with this information.  I have no notion of where these requests are coming from since no other information is included here.  Is there anything else I can investigate?  The number 11 doesn't seem to line up with anything I can think of either, there are 3 searchheads, 3 indexers, 1 cluster manager, in this particular deployment.  Not sure where the 11 requests is coming from. ... View more

I have old searchheads that were removed via "splunk remove shcluster-member" command.  They rightfully do not show when I run "splunk show shcluster-status", however when I run  "splunk show kvstore-status" all the removed searchheads still show in this listing.  How do I get them removed from the kvstore clustering as well? ... View more

For the past couple weeks I will at least once per day have one of our indexers go into internal logs only mode, and the reason it states is that License is expired.  It's a bogus message since the license definitely is not expired and also not even close to exceeded, and restarting splunk service on the indexer always clears the error.  Unfortunately not much more is provided by the splunk logs that would indicate anything I can investigate. Has anyone ever ran into similar, or might know where I can look to troubleshoot this further?  It's making my life pretty tough because I have to constantly be restarting indexers due to this error. ... View more

I wasn't sure if having multiple different license managers would cause any violations.  Ideally we really do not like the idea of having a single point of failure for our license manager, and are looking to implement redundancy.  Is this possible or will it cause issues? ... View more

I have been investigating a particular search an api user runs which has become markedly slower past a specific date.  When looking in the audittrail internal logs, what I noticed is that there is no significant increase in event count, however the "total_slices" number significantly increases from before the date through after the date. I couldn't find much information in the documentation on what this value represents.  Does this mean the data within each event increased around that time? ... View more

I have a sourcetype that is exhibiting very odd behavior.  If I try to run a lookup command such as the following: index=index_here sourcetype=sourcetype_here |lookup lookup_name JoiningID as JoiningID output Value1 Value2 It will not give me Value1 or Value2 in my results, however if I instead run: index=index_here sourcetype=sourcetype_here |join type=left JoiningID [|inputlookup lookup_name] I get the Value1 and Value2 here joined in no problem.  What are some reasons for the actual lookup command not giving me any values? ... View more

In the splunkd.log I can see the following:  "Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured." If I check the kvstore-status it just says the kvstore status is down for this new member.  The normal shcluster-status shows as UP and the kvstore as ready however for this new member.  Not sure what I can do to try and force the kvstore to initialize?  I have tried shutting splunk down on the new member and doing a kvstore-clean and restarting but it still isn't taking. Splunk version 9.0.0   Any thoughts on what else I can try? ... View more

I have some sources that are coming in as json, and I am experiencing odd behavior where I cannot search on a particular field, but I can only find the value when doing a search against the _raw data. So for example, I have a field let's say "cluster", and I see it is also extracted just fine in the "Interesting fields" on the lefthand side.  One of the values we'll say is "cluster-name-A".   If I search in the query bar for:     cluster="cluster-name-A" sourcetype=mysourcetype index=myindex     I get no results, however if I just do a blanket search:     cluster-name-A sourcetype=mysourcetype index=myindex     My expected results come back fine. What can I investigate here to see why it will not let me use the fieldname in our searches? ... View more

Had to take an indexer down for several days while a SSD was replaced, I used the "splunk offline --enforce-counts" command to allow the data to replicate back out to the other indexers (we have replication factor of 1).  I'm curious now after the SSD has been replaced, what is the best option to rejoin this host back to the cluster? ... View more

We have a team that are sending far too many wasteful logs to us for a specific sourcetype.  It's going to take them a while to tune their logging, and I was wondering if there is a way short of invalidating their token that I could just deny one specific sourcetype from being ingested? ... View more

I keep getting an error message in our messages section at the top, stating that Search head cluster member ____ is having problems pulling configurations from the search head cluster captain.  Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member.    I've performed destructive resync several times, and also when I use the show kvstore status function from any searchhead including the supposedly affected one it claims there are no actual issues with the searchhead, it just says it is a non-captain KV store member just like the rest.  If I try to manually close out the error message it just comes back. What else can I look into here? ... View more