The numbers are not exact, from the DS Forwarder Management > 1275, dc(h) from metrics > 1287, and the total stats count from the final query > 1166  so its not accurate.  I will need to create a lookup of UFs. Thank you for your support. ... View more

      index=_internal source=*license_usage.log earliest=-1d@d latest=now [search index=_internal source=*metrics.log fwdType=uf earliest=-1d@d latest=now | rename hostname as h | fields h] | stats sum(b) as total_usage_bytes by h | eval total_usage_gb = round(total_usage_bytes/1024/1024/1024, 2) | fields - total_usage_bytes | addcoltotals label="Total" labelfield="h" total_usage_gb   I think this is what I wanted, unless someone thinks its inaccurate?  Please advise. TY   ... View more

Thank you for the reply.  I also looked at this log but it requires curating an exact list of the UFs, bc I have some pollution, e.g. h= HFs, SC4S, etc.  The license_usage log may be the best route if I can put together a lookup of just UFs. ... View more

Thank you for your reply, do you have a method of querying to get an answer for my question? I am not finding the key logs containing UF data thruput or ingest information.   ... View more

yes thx I accepted that over a year ago ... View more

Curious, is TZ=GB for UK valid or did I misread something? ... View more

POA 1) disable KVstore on the indexer instances only 2) update KVstore all other Splunk instances with a "ready" enabled KVstore to be safe.  **In my case, all instances are default enabled ** Hopefully everything goes well! I will post if it goes sideways. ... View more

Well, in line with your recommendations... I am being advised by Internal Splunk Personnel, to upgrade everything except indexers... My feeling is to disable the kvstore on the indexers but not the other nodes like IDXcluster master etc... Would you agree? (you can conclude that I value your opinion with equal weight as support 😉, maybe more)   ... View more

Thanks for the clarification (and persevering thru my questions).  I really do appreciate it! My concerns, re: IDXCluster and KVstore, are due to all the non-sensical configurations the previous admins left for me.... It would not surprise me to see something unusual here, as I have seen some really goofy stuff.   For instance , I am concerned they did some sort of remote kvstore replication/sync to the indexer peers etc... maybe that doesn't matter, not 100% about that. Would you advise that I "disable" the KVstore on the indexers? for safety? per Splunk docs>>> https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/AboutKVstore   RE: "upgrade only Search Heads and Heavy Forwarders" ,  I see collections (probably default) on my MC/LM instance and Deployment Server instance as well.  So I am thinking upgrade those as well? IDK. when I run this on my MC/LM instance >>>        | rest splunk_server="local" "/services/search/distributed/peers/" | search disabled=0 | rename peerName AS splunk_server | fields host splunk_server | join type=left splunk_server [| rest splunk_server="*" "/services/kvstore/status" | table splunk_server current.status current.replicationStatus current.storageEngine ] | sort - current.storageEngine         I see every instance (including indexers)  has a default enabled kvstore in "ready"...  So now, knowing this information, does that change your advice? RE: support, IDK, I do feel support should be able to answer these questions.  I get support is "break/fix" but do I really have to break it first to get help or a couple definitive answers? I am sure Support will kick it to our Sales Rep and they will say we need to engage PS for 2 week$ 🤣...   Thank you!   ... View more

Thank you for the response.  You answered me before in another post >> https://community.splunk.com/t5/Deployment-Architecture/In-a-distributed-and-clustered-environ-which-kvstores-need/td-p/633203  upgrade all nodes.  So I presume there is no harm upgrading  a kvstore engine and version on an instance w/ or w/out any active collections? I am scrambled and support won't even give me a definite answer. ... View more