yes thx I accepted that over a year ago ... View more

Curious, is TZ=GB for UK valid or did I misread something? ... View more

POA 1) disable KVstore on the indexer instances only 2) update KVstore all other Splunk instances with a "ready" enabled KVstore to be safe.  **In my case, all instances are default enabled ** Hopefully everything goes well! I will post if it goes sideways. ... View more

Well, in line with your recommendations... I am being advised by Internal Splunk Personnel, to upgrade everything except indexers... My feeling is to disable the kvstore on the indexers but not the other nodes like IDXcluster master etc... Would you agree? (you can conclude that I value your opinion with equal weight as support 😉, maybe more)   ... View more

Thanks for the clarification (and persevering thru my questions).  I really do appreciate it! My concerns, re: IDXCluster and KVstore, are due to all the non-sensical configurations the previous admins left for me.... It would not surprise me to see something unusual here, as I have seen some really goofy stuff.   For instance , I am concerned they did some sort of remote kvstore replication/sync to the indexer peers etc... maybe that doesn't matter, not 100% about that. Would you advise that I "disable" the KVstore on the indexers? for safety? per Splunk docs>>> https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/AboutKVstore   RE: "upgrade only Search Heads and Heavy Forwarders" ,  I see collections (probably default) on my MC/LM instance and Deployment Server instance as well.  So I am thinking upgrade those as well? IDK. when I run this on my MC/LM instance >>>        | rest splunk_server="local" "/services/search/distributed/peers/" | search disabled=0 | rename peerName AS splunk_server | fields host splunk_server | join type=left splunk_server [| rest splunk_server="*" "/services/kvstore/status" | table splunk_server current.status current.replicationStatus current.storageEngine ] | sort - current.storageEngine         I see every instance (including indexers)  has a default enabled kvstore in "ready"...  So now, knowing this information, does that change your advice? RE: support, IDK, I do feel support should be able to answer these questions.  I get support is "break/fix" but do I really have to break it first to get help or a couple definitive answers? I am sure Support will kick it to our Sales Rep and they will say we need to engage PS for 2 week$ 🤣...   Thank you!   ... View more

Thank you for the response.  You answered me before in another post >> https://community.splunk.com/t5/Deployment-Architecture/In-a-distributed-and-clustered-environ-which-kvstores-need/td-p/633203  upgrade all nodes.  So I presume there is no harm upgrading  a kvstore engine and version on an instance w/ or w/out any active collections? I am scrambled and support won't even give me a definite answer. ... View more

I think, better safe than sorry, looks like some apps use KVstores...  ... View more

Well I understand your point about "this"... but that's the problem, I couldn't find an error with the skipped searches... unless I am missing something. Since I did the rolling restart (reset) there are no more skipped searches. Previously I looked for the longest running searches and none were over-running their schedules, that I could see.  For example one search took an hour approx., but it ran every 4 hours.   Since I did some optimizing there were only 3 scheduled searches that produced the warning which I identified with    index="_internal" sourcetype="scheduler" | eval scheduled=strftime(scheduled_time, "%Y-%m-%d %H:%M:%S") | stats values(scheduled) as scheduled values(savedsearch_name) as search_name values(status) as status values(reason) as reason values(run_time) as run_time values(dm_node) as dm_node values(sid) as sid by _time,savedsearch_name | sort -scheduled | table scheduled, search_name, status, reason, run_time     When I looked back at those 3 specific searches, they were not over-running the schedules, so I was wondering how it got stuck thinking it was "piling up" vs "still running".      I am trying to understand/investigate, if a search is "skipped" then when the shc scheduler retries that previously skipped search at its next runtime, "how can I see that the shc CPT thinks its still running"?  And looking back at the "skipped" events, they don't contain "run_time"...  so I look back historically to find a day with a high value.  But when the searches were running they took max 4 seconds with avg of 2 seconds to complete, which is why I thought the scheduled searches were piling up.  Hope that makes sense. The only other variable I can think of is that these searches are using the "| dbxquery" cmd from Splunk DB Connect app. So did it the SHC just get stuck? Any further thoughts appreciated. TY     ... View more

Apparently this works (for some hosts) so cannot say for sure the others have logs for this input path or the splunkd user is different. [monitor://$ExchangeInstallPath\TransportRoles\Logs\FrontEnd\AgentLog]           ... View more

the conf-inspector has less functionality https://splunkbase.splunk.com/app/3957 ... View more

@gcusello  Thank you for the reply and suggestion. I agree with you about replication, however in my case, the shc was not replicating correctly.  The previous admin did not know what he was doing and others were restarting shc nodes individually causing corruption. Specifically I was seeing transforms in the search app that were not replicating correctly.  As well the lookups were different across the shc nodes. Its a real mess! The PS assessment app has a conf checker but it is not working correctly and is probably missing a dependency.    I will try you suggestion and get back to you.  TY! ... View more

The reason I asked this is bc someone posted videos showing to use the encrypted pass4Symm key, rather than the clear. ... View more

I don't know if this is the correct method, but it seems to have worked. Using "find" command, I found the scheduled search under the /opt/splunk/etc/users/<user-name>/<app-name>  in savedsearches.conf.  Then I went in to each shc node and disabled it, then did a rolling restart.     Interestingly, under /opt/splunk/etc/users/<user-name>/<app-name>/metadata >in local.meta there was nothing for the owner, completely missing... but the search name was in there.   I have no idea how the shc got this way, but would really like to know, if anyone can explain.   Thank you   ... View more