Hi we have list of hosts that are logging splunk and sending logs to splunk .Since when i created the lookup to check whether they are logging splunk .I found wrong results .In below query iam trying to get results that hosts that are not sending logs to splunk .
|inputlookup samplehostsrecentlist.csv | join type=left host [|tstats count by host] | fillnull value=0 count | search count=0 |fields host os pci count
After searching results i found that host that is in list shows sending logs to splunk .Means something wrong with search also when i use metadata search to get results that also shows wrong results.
Q:We have 400 hosts list and i created csv file with columns host ,os,pci ,ip .goal is to see whether which hosts are not sending logs to splunk .Also whether it is phone homing splunk deployment server ,when was last time we get events .Please help with query which gives accurate results .
I assume that there is something wrong with search or does splunk gives wrong results for even last 24 hour results for when checking more hosts ? i believe splunk gives accurate results .
The larger question is 'do you want to know when the a machine hasn't sent any information to splunk for a speicifc data set OR do you want to know when no data is being sent?'
the prior is much more difficult to answer, however for the latter, I'd recommend using metadata. The Splunk DMC has a pretty good query for that you can use. It's a dashboard view.
So there's a few ways to do this, and i'll share one of my processes.
Check the Deployment Server
Have a standard deployment of packages to go out to a machine
index=os host=<myhost>will tell you if you're receiving data from that host or not.
for all hosts, in all indexes, I'd suggest setting up a scheduled search to run periodically dumping the 'host' names reporting to a .csv file with |outputlookup, and then do a couple
|inputlookup commands with
Here's an example of what I mean:
I've used this one in the past:
| metadata type=hosts index!=summary* | where (now()-recentTime<7200) | join host [|metadata type=hosts index=_internal | where (now()-recentTime<7200) | stats count by host recentTime] | stats count by host recentTime | rex field=host "(?P<hostname>[^.]+)" | eval hostname=lower(hostname) | eval sourceHost=hostname | eval connectionType="universal forwarder" | eval arch="undefined" | eval lastReceived = recentTime | eval lastConnected=recentTime | eval KB = round(1000, 4) | eval eps = round(100, 4) | eval mystatus = if(lastConnected<(now()-100),"quiet","dunno") | eval status = if(lastConnected<(now()-300),"missing",if(mystatus="quiet","quiet","active")) | stats count(KB) as Count by sourceHost connectionType lastConnected lastReceived status KB eps | fields sourceHost connectionType lastConnected lastReceived status KB | rename sourceHost as Forwarder destPort as "Destination Port" connectionType as "Forwarder Type" version as "Splunk Version" status as "Current Status" lastConnected as "Last Connected" lastReceived as "Last Data Received" KB as "Total KB" eps as "Average Events Per Second" | fieldformat "Last Connected"=strftime('Last Connected', "%D %H:%M:%S %p") | fieldformat "Last Data Received"=strftime('Last Data Received', "%D %H:%M:%S %p")
@splunker969, refer the Splunk Documentation on a similar topic.