@livehybrid not working, its giving 0 events. I am trying to list those forwarders which are disconnected from DS ... View more

Hi, I am working in a project where we have multiple splunk on-prem instances (mostly 3-4) I am trying to get a list of all the forwarders which are currently disconnected to Deployment server. I cannot login to the backend and the UI of Deployemnt server forwarder management doesn't show any details (may be because it is acting as a Load balancer) for one of the splunk instance. So I used the below query to list all the active forwarders - index=_internal source=metrics.log splunk_server IN (ncepnspidxdb*, ncwpnspidxdb*) group=tcpin_connections fwdType=uf | stats count by hostname | fields hostname and , to list all the connected forwarders to DS -  index=_internal sourcetype=splunkd component=PubSubSvr "*handshake*" OR "*reply*" splunk_server IN (ncepnspidxdb*, ncwpnspidxdb*) | rex "\/handshake\/reply\/(?P<DeploymentClient>[^\/]+)" | stats count by DeploymentClient | rename DeploymentClient as host | fields - count    I have tried multiple ways using search A | search NOT [search B] , join left and what not. Nothing is working. Could anyone please help me with a query that will give me the list of forwarders which are disconnected to DS. ... View more

  @kknairr  Exactly, what I have observed. But the issue is for some reason this rest query doesn't return any value for 2 of the Splunk on prem instances when I run in UI of DS under search & reporting. Also, those two doesn't show any forwarders phoning home under forwarder management (main DS is acting as a LB as far I am aware) I even tried using the query like this in search head -  | rest /services/deployment/server/clients splunk_server="dedicated_splunk_server_for _that_instance" but it doesn't work So, I took a different approach, I am using an internal index query to get all the forwarders available using the query - index=_internal source=*metrics.log* group=tcpin_connections os=* fwd_type=uf |stats latest(fwdType) AS forwarder_type latest(os) AS os latest(version) AS version by hostname and to get the connected hosts to DS -  index=_internal sourcetype=splunkd component=PubSubSvr | rex "\/handshake\/reply\/(?P<DeploymentClient>[^\/]+)" | stats count by host DeploymentClient | rename host as DeploymentServer | fields - count   ... View more

@kknairr  I also observed that this rest query is only working when I run this under UI of my DS under search and reporting app, however in my client environment, we have multiple splunk (4-5) on prem instances whose search head could communicate with each other. So if I run this query with - | rest splunk_server=local /services/deployment/server/clients on any of the Search head, it does either return any value and without splunk_server=local it tends to pull hosts which doesn't belong to that particular environment. In such a case what is the best way I could get the disconnected host? ... View more

@isoutamo  What I am trying to achieve here is - I am trying to list down all the universal forwarders which are currently disconnected or not communicating to its Deployment Server using any idea way which ever works. P.S. We are using splunk on-prem instances I have access to the UI as well as backend of the Deployment server. So any approach would work for me!! ... View more

@kknairr I am using the similar logic where I am - 1. Calculating the last time a forwarder has phone homed/checked in 2. Converting the last-seen in hours 3. Filtering forwarder hasn’t contacted DS for 10+ hours → might be offline, blocked, or misconfigured. | rest /services/deployment/server/clients | eval lastSeenHours = (now() - lastPhoneHomeTime) / 3600 | where lastSeenHours > 10 | eval lastPhoneHomeReadable = strftime(lastPhoneHomeTime, "%Y-%m-%d %H:%M:%S") | table hostname ip lastPhoneHomeReadable lastSeenHours | sort - lastSeenHours | rest /services/deployment/server/clients | eval lastSeenHours = (now() - lastPhoneHomeTime) / 3600 | where lastSeenHours > 10 | eval lastPhoneHomeReadable = strftime(lastPhoneHomeTime, "%Y-%m-%d %H:%M:%S") | table hostname ip lastPhoneHomeReadable lastSeenHours | sort - lastSeenHours But one doubt here is that this output gives agent: in error under forwarder tab of forwarder management and not agent: offline Also the count for agent: offline keeps changing. Any thoughts on it ... View more

@kknairr  I am using the similar logic where I am - 1. Calculating the last time a forwarder has phone homed/checked in 2. Converting the last-seen in hours 3. Filtering forwarder hasn’t contacted DS for 10+ hours → might be offline, blocked, or misconfigured. | rest /services/deployment/server/clients | eval lastSeenHours = (now() - lastPhoneHomeTime) / 3600 | where lastSeenHours > 10 | eval lastPhoneHomeReadable = strftime(lastPhoneHomeTime, "%Y-%m-%d %H:%M:%S") | table hostname ip lastPhoneHomeReadable lastSeenHours | sort - lastSeenHours But one doubt here is that this output gives agent: in error under forwarder tab of forwarder management and not agent: offline Also the count for agent: offline keeps changing. Any thoughts on it ... View more

@PickleRick  Understood! Just one last confirmation. For a host, I observe that after every 12 seconds, the host is trying to establish handshake with the Deployment server and lastly it says "handshake done" Does that means it was finally able to establish connection with the DS??     ... View more

Hi @PickleRick  I was able to get the correct working query. index=_internal sourcetype=splunkd component=DC:DeploymentClient splunk_server=* "*phonehome*" OR "*handshake*" OR "not_connected" | stats latest(_time) AS lastTime BY host | eval age=round((now() - lastTime)/3600, 1) | where age >=24 | eval lastTime=strftime(lastTime, "%b %d, %Y %H:%M:%S") | table host age lastTime | sort 0 - age Apparently I was overthinking it 😅 ... View more

@PickleRick  We have HF as well in my instance, so if I do -  | tstats count where index=_internal by host | dedup hostname | table hostname Is it not going to return HF as well?  ... View more

@PickleRick  I am still not able to figure this out. I am trying to get a list of the all the ufs which are disconnected or not communicating to the Deployment server. I am breaking it down in this way . 1. To List all the existing forwarders for my instance , and  2. List all the forwarders which are establishing connection with DS Hence, Missing/Disconnect UFs = All - (active UFs) In my monitoring console, under Forwarder > Deployment, I see data which shows all the active and missing forwarders, which is pulling its data from a lookup called "| inputlookup dmc_forwarder_assets" , like you said earlier. So, if I want to get all the forwarders which are there in my environment and is active, I am using this query below- | inputlookup dmc_forwarder_assets | search forwarder_type="uf" AND status="active" | dedup hostname | table hostname   And to get list of forwarders I am using the below query - index=_internal sourcetype=splunkd component=DC:HandshakeReplyHandler | dedup host | table host     As, in my instance , under internal logs, only 3 values for the field component seems relevant - DC:HandshakeReplyHandler, DC:DeploymentClient , DC:PhonehomeThread and DS_DC_Common So, with my understanding I used DC:HandshakeReplyHandler as it gives the message - Handshake done (which basically means that uf was able to establish connection with the DS ) Am I going in right direction?? Please reply ... View more

@PickleRick  To get to this "forwarder inventory database built in Monitoring Console." Is this the correct steps  Monitoring Console → Forwarders → Deployment  ??   ... View more