I presume those are all ansible plays. Thank you. Now to run those and see how they _will_ fail.  ... View more

Finally getting around to this.   On all servers where splunk is installed the service splunkd is running.   (an example from grep: {server}-106 | CHANGED | rc=0 >> splunk 5131 1 0 Apr07 ? 01:18:04 splunkd -p 8089 start splunk 5233 5131 0 Apr07 ? 00:00:00 [splunkd pid=5131] splunkd -p 8089 start [process-runner] )   running the index=_internal host!=utility-log-* | stats max(_time) as _time count by host returns many servers.   Again. The only changes made to the clients and servers has been patching, updating and rebooting . Yes _some_ of the servers did not have the imitating with accept-license. but that's been fixed.   So it looks like I have no hope of getting this fixed due to the way Splunk does "support". ... View more

next step.   ran another oneliner to check splunk status While there are a few which do not have license accepted, which I will fix shortly, all others are running properly, and the log shows current activity. ... View more

Ran an ansible one-liner against all my clients. All are running splunkd.   Will check the next suggestion shortly.   (work got busy) ... View more

1 - probably 2 - if by that you mean /opt/splunk/splunkforwarder, yes that is the default on all clients in our environment 3 - I see almost all of our servers using the search given 4 - haven't touched our config files since installation. I have done splunk updates and OS patching. Both using a shutdown/patch-or-update/restart sequence that has been approved directly by splunk.   I expect I will get nowhere here as the answers so far have presumed knowledge that the admin team here was _never_ _given_.  Again. We were supposed to have training on days 4 and 5 of installation. But since days 1 and 2 were taken doing tasks that we were told had to be done before install could happen - even though we asked what to do before the installation - we DID NOT GET TRAINING. I know the very basics. But nothing more. ... View more

huh. on admin nodes I get either a reading error or a message about 0 population, but splunk is running on all splunk servers. ... View more