I had to rebuild a searchhead some time back, everything works fine on the searchhead and it functions just fine within the searchhead cluster itself, however when I try to view the status of my searchheads via the Index Cluster Manager UI, this rebuilt searchhead always shows as "Down".  I am pretty certain it's because when I check the GUID locally on the searchhead listed here:   /opt/splunk/etc/instance.cfg It does not match the GUID the Index Cluster Manager has associated with that searchhead.  I cannot for the life of me figure out how to get the Cluster Manager to pick up the new guid though, does anyone know how I can accomplish this? ... View more

I have been tasked with building out new instances of anything that runs an older OS, and for our EC2 instances this most often means I have to rebuild the instance on Amazon Linux 2023.  What I've noticed, particularly with our heavy forwarders, is that they are constantly dying and ballooning memory to 100% with no rhyme or reason as to why (not heavy load, happens to random servers at random times, and there should be no world where the memory even comes close to getting consumed considering the low level of traffic for the servers in question which are in our lab environment).   We're running splunk version 10.0.2 Nothing helpful ever in the logs aside from indications that it ran OOM.  I've randomly tested with making absurdly large instances to see if the issue still occurs and sure enough it does, there's not enough memory in the world to handle whatever error seems to be occurring, so it seems to be something more akin to a bug that's occurring. I'm curious if anyone else has ran into issues with AL2023 and running Splunk? ... View more

Lately our searchheads will run into issues where the srtemp folder baloons to 80+GB and fills the local hard drive.  To temporarily fix, I have to shut down splunk and then run the splunk command: clean-srtemp  It will then be good for several days to a week and eventually it's back again, rinse and repeat.  I'm curious what my options are for determining what is causing the srtemp folder to fill consistently? ... View more

This is an odd one happening on each of our indexers.  The same behavior happens quite frequently, where we will get exactly 11 of these Remote token requests from splunk-system-user, and exactly 1 of them will fail.  Here is how it looks in the audit logs. 04-22-2024 21:30:31.964 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:31.964, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:31.986 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:31.986, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:32.384 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:32.384, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:32.395 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:32.395, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:40.687 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:40.687, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:40.694 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:40.694, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:46.803 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:46.803, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:46.815 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:46.815, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:47.526 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:47.526, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:47.542 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:47.542, user=splunk-system-user, action=Remote token requested, info=success] 04-22-2024 21:30:55.317 -0700 INFO  AuditLogger - Audit:[timestamp=04-22-2024 21:30:55.317, user=splunk-system-user, action=Remote token requested, info=failed] My problem is I can't do much more with this information.  I have no notion of where these requests are coming from since no other information is included here.  Is there anything else I can investigate?  The number 11 doesn't seem to line up with anything I can think of either, there are 3 searchheads, 3 indexers, 1 cluster manager, in this particular deployment.  Not sure where the 11 requests is coming from. ... View more

I have old searchheads that were removed via "splunk remove shcluster-member" command.  They rightfully do not show when I run "splunk show shcluster-status", however when I run  "splunk show kvstore-status" all the removed searchheads still show in this listing.  How do I get them removed from the kvstore clustering as well? ... View more

For the past couple weeks I will at least once per day have one of our indexers go into internal logs only mode, and the reason it states is that License is expired.  It's a bogus message since the license definitely is not expired and also not even close to exceeded, and restarting splunk service on the indexer always clears the error.  Unfortunately not much more is provided by the splunk logs that would indicate anything I can investigate. Has anyone ever ran into similar, or might know where I can look to troubleshoot this further?  It's making my life pretty tough because I have to constantly be restarting indexers due to this error. ... View more

I wasn't sure if having multiple different license managers would cause any violations.  Ideally we really do not like the idea of having a single point of failure for our license manager, and are looking to implement redundancy.  Is this possible or will it cause issues? ... View more

I have been investigating a particular search an api user runs which has become markedly slower past a specific date.  When looking in the audittrail internal logs, what I noticed is that there is no significant increase in event count, however the "total_slices" number significantly increases from before the date through after the date. I couldn't find much information in the documentation on what this value represents.  Does this mean the data within each event increased around that time? ... View more

I have a sourcetype that is exhibiting very odd behavior.  If I try to run a lookup command such as the following: index=index_here sourcetype=sourcetype_here |lookup lookup_name JoiningID as JoiningID output Value1 Value2 It will not give me Value1 or Value2 in my results, however if I instead run: index=index_here sourcetype=sourcetype_here |join type=left JoiningID [|inputlookup lookup_name] I get the Value1 and Value2 here joined in no problem.  What are some reasons for the actual lookup command not giving me any values? ... View more

In the splunkd.log I can see the following:  "Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured." If I check the kvstore-status it just says the kvstore status is down for this new member.  The normal shcluster-status shows as UP and the kvstore as ready however for this new member.  Not sure what I can do to try and force the kvstore to initialize?  I have tried shutting splunk down on the new member and doing a kvstore-clean and restarting but it still isn't taking. Splunk version 9.0.0   Any thoughts on what else I can try? ... View more

I have some sources that are coming in as json, and I am experiencing odd behavior where I cannot search on a particular field, but I can only find the value when doing a search against the _raw data. So for example, I have a field let's say "cluster", and I see it is also extracted just fine in the "Interesting fields" on the lefthand side.  One of the values we'll say is "cluster-name-A".   If I search in the query bar for:     cluster="cluster-name-A" sourcetype=mysourcetype index=myindex     I get no results, however if I just do a blanket search:     cluster-name-A sourcetype=mysourcetype index=myindex     My expected results come back fine. What can I investigate here to see why it will not let me use the fieldname in our searches? ... View more

Had to take an indexer down for several days while a SSD was replaced, I used the "splunk offline --enforce-counts" command to allow the data to replicate back out to the other indexers (we have replication factor of 1).  I'm curious now after the SSD has been replaced, what is the best option to rejoin this host back to the cluster? ... View more

We have a team that are sending far too many wasteful logs to us for a specific sourcetype.  It's going to take them a while to tune their logging, and I was wondering if there is a way short of invalidating their token that I could just deny one specific sourcetype from being ingested? ... View more

I keep getting an error message in our messages section at the top, stating that Search head cluster member ____ is having problems pulling configurations from the search head cluster captain.  Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member.    I've performed destructive resync several times, and also when I use the show kvstore status function from any searchhead including the supposedly affected one it claims there are no actual issues with the searchhead, it just says it is a non-captain KV store member just like the rest.  If I try to manually close out the error message it just comes back. What else can I look into here? ... View more

Encountering a very odd issue where I have a daily summary index that has pretty simple key=value pairings for fields, but I can no longer search on the fields specifically.  For instance an event might have a field included that reads cluster=cluster1A and if I search for cluster=cluster1A, I get no results, but if I search for just the text cluster1A, I get results.  What might I be able to look into here? ... View more

We have a single searchhead that continually fills up the dispatch directory.  I manually have to go in and clear it out.  There's nothing special about this searchhead and I've even gone so far as to take it out of our load balancer such that only scheduled searches are being ran against it, yet still the dispatch directory continually ends up filling back up. What could be some areas I can look into to try and troubleshoot why a single searchhead would have such trouble clearing out its dispatch directory? ... View more