How can I write a search to find hosts that are not sending logs to Splunk or show the last time a host sent logs to Splunk?

[splunker969]

Hi we have list of hosts that are logging splunk and sending logs to splunk .Since when i created the lookup to check whether they are logging splunk .I found wrong results .In below query iam trying to get results that hosts that are not sending logs to splunk .
query -

|inputlookup samplehostsrecentlist.csv | join type=left host [|tstats count by host] | fillnull value=0 count | search count=0 |fields host os pci count

After searching results i found that host that is in list shows sending logs to splunk .Means something wrong with search also when i use metadata search to get results that also shows wrong results.

Q:We have 400 hosts list and i created csv file with columns host ,os,pci ,ip .goal is to see whether which hosts are not sending logs to splunk .Also whether it is phone homing splunk deployment server ,when was last time we get events .Please help with query which gives accurate results .
I assume that there is something wrong with search or does splunk gives wrong results for even last 24 hour results for when checking more hosts ? i believe splunk gives accurate results .

[tmarlette]

The larger question is 'do you want to know when the a machine hasn't sent any information to splunk for a speicifc data set OR do you want to know when no data is being sent?'

the prior is much more difficult to answer, however for the latter, I'd recommend using metadata. The Splunk DMC has a pretty good query for that you can use. It's a dashboard view.

[splunker969]

Goal is to check if hosts in not sending logs to splunk then we need to install agent on those hosts and trouble shoot on host ig there is agent and not sending logs .

[tmarlette]

So there's a few ways to do this, and i'll share one of my processes.

1. Check the Deployment Server

   • If the DS doesn't have the machine listed with the host name in question, then either a forwarder isn't installed or the deploymentclient.conf isn't configured correctly.

2. Have a standard deployment of packages to go out to a machine

   • If you always send out the windows TA to windows boxes, and Linux TA to Linux boxes, then just have that in your serverclass.conf configured. That way, whenever a forwarder is installed, it starts collecting data right away and a simply index=perfmon host=<myhost> or index=os host=<myhost> will tell you if you're receiving data from that host or not.

for all hosts, in all indexes, I'd suggest setting up a scheduled search to run periodically dumping the 'host' names reporting to a .csv file with |outputlookup, and then do a couple |inputlookup commands with |set diff.

Here's an example of what I mean:
https://answers.splunk.com/answers/56586/list-difference-between-two-csv-files.html

[splunker969]

Thanks @tmarlette

[splunker969]

Hi tmarlette ,
Can you help me with search that will give accurate results .
Goal is to get hosts names that are not logging splunk and not phone homing with deployment server ?

[tmarlette]

I've used this one in the past:

| metadata type=hosts index!=summary*
| where (now()-recentTime<7200) 
| join host [|metadata type=hosts index=_internal 
| where (now()-recentTime<7200) 
| stats count by host recentTime]
| stats count by host recentTime
| rex field=host "(?P<hostname>[^.]+)" 
| eval hostname=lower(hostname)
| eval sourceHost=hostname
| eval connectionType="universal forwarder"
| eval arch="undefined"
| eval lastReceived = recentTime
| eval lastConnected=recentTime
| eval KB = round(1000, 4)
| eval eps = round(100, 4)
| eval mystatus = if(lastConnected<(now()-100),"quiet","dunno")
| eval status = if(lastConnected<(now()-300),"missing",if(mystatus="quiet","quiet","active"))
| stats count(KB) as Count by sourceHost connectionType lastConnected lastReceived status KB eps 
| fields sourceHost connectionType lastConnected lastReceived status KB 
| rename sourceHost as Forwarder destPort as "Destination Port" connectionType as  "Forwarder Type" version as "Splunk Version" status as "Current Status" lastConnected as "Last Connected" lastReceived as "Last Data Received" KB as "Total KB" eps as "Average Events Per Second"
| fieldformat "Last Connected"=strftime('Last Connected', "%D %H:%M:%S %p")
| fieldformat "Last Data Received"=strftime('Last Data Received', "%D %H:%M:%S %p")

[splunker969]

Thanks tmarlette

[niketn]

@splunker969, refer the Splunk Documentation on a similar topic.

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Addinfo#2._Determine_which_heart...

[splunker969]

Thanks @nikenilay 🙂