In the splunkd.log I can see the following:  "Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured." If I check the kvstore-status it just says the kvstore status is down for this new member.  The normal shcluster-status shows as UP and the kvstore as ready however for this new member.  Not sure what I can do to try and force the kvstore to initialize?  I have tried shutting splunk down on the new member and doing a kvstore-clean and restarting but it still isn't taking. Splunk version 9.0.0   Any thoughts on what else I can try? ... View more

I have some sources that are coming in as json, and I am experiencing odd behavior where I cannot search on a particular field, but I can only find the value when doing a search against the _raw data. So for example, I have a field let's say "cluster", and I see it is also extracted just fine in the "Interesting fields" on the lefthand side.  One of the values we'll say is "cluster-name-A".   If I search in the query bar for:     cluster="cluster-name-A" sourcetype=mysourcetype index=myindex     I get no results, however if I just do a blanket search:     cluster-name-A sourcetype=mysourcetype index=myindex     My expected results come back fine. What can I investigate here to see why it will not let me use the fieldname in our searches? ... View more

Had to take an indexer down for several days while a SSD was replaced, I used the "splunk offline --enforce-counts" command to allow the data to replicate back out to the other indexers (we have replication factor of 1).  I'm curious now after the SSD has been replaced, what is the best option to rejoin this host back to the cluster? ... View more

We have a team that are sending far too many wasteful logs to us for a specific sourcetype.  It's going to take them a while to tune their logging, and I was wondering if there is a way short of invalidating their token that I could just deny one specific sourcetype from being ingested? ... View more

I keep getting an error message in our messages section at the top, stating that Search head cluster member ____ is having problems pulling configurations from the search head cluster captain.  Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member.    I've performed destructive resync several times, and also when I use the show kvstore status function from any searchhead including the supposedly affected one it claims there are no actual issues with the searchhead, it just says it is a non-captain KV store member just like the rest.  If I try to manually close out the error message it just comes back. What else can I look into here? ... View more

Encountering a very odd issue where I have a daily summary index that has pretty simple key=value pairings for fields, but I can no longer search on the fields specifically.  For instance an event might have a field included that reads cluster=cluster1A and if I search for cluster=cluster1A, I get no results, but if I search for just the text cluster1A, I get results.  What might I be able to look into here? ... View more

We have a single searchhead that continually fills up the dispatch directory.  I manually have to go in and clear it out.  There's nothing special about this searchhead and I've even gone so far as to take it out of our load balancer such that only scheduled searches are being ran against it, yet still the dispatch directory continually ends up filling back up. What could be some areas I can look into to try and troubleshoot why a single searchhead would have such trouble clearing out its dispatch directory? ... View more

I have a lookup that recently stopped auto extracting fields. What I've noticed is that if I do a join, I can join if in the subsearch I specifically search for that row, but doing the normal lookup command gives me nothing. For example something like: index=a sourcetype=a host=host1 | lookup host_lookup host as host output fieldA Does not give me fieldA value for host1, however if I do: index=a sourcetype=a host=host1 | join host [|inputlookup host_lookup | table host fieldA| search host=host1] I get fieldA just fine in that case. So clearly it would appear to me some sort of limit is getting hit, even though I don't seem to be seeing any indication in the ui or Job inspection stating me that I am hitting a limit. Does anyone know if this is indeed a limit I'm hitting? Or is there anything else I can look into? ... View more

I was hoping for an app I'm developing that a user could simply fill in a field for where certain sourcetypes for the app will be ingested from, because it will vary from user to user depending on where they store the data my app ingests locally. Is there a way I can configure this in setup.xml? Or do they always need to manually configure this in their inputs.conf? ... View more

I have an index I'm using to backfill a bunch of data, and as I'm tracking the event count by sources, I'm seeing splunk throw away events literally by the millions randomly (I'll keep track of the events of one of my sources, then I'll check again 5 minutes later and the number is over a million less than it used to be.). None of the limits set should be getting hit. The only thing being hit is the warm path but that should just be rolling into the much larger allotment I gave for Cold, and Cold isn't even near being full, yet I'm getting events thrown out left and right. What can I look into here? I'm having trouble trusting the integrity of my data when I see the event counts moving backwards even though Cold Path isn't even close to being filled up, and data age isn't close to being hit either. ... View more