I am relatively new to a company that has used Splunk Professional Services to spin up a Splunk Cloud environment before I was hired. The company IT has onboarded a lot of AWS, Azure, on-prem and network devices so far. I’m trying to verify that they are in fact sending logs into the Splunk index so that I can eventually apply use cases and alerting on the logs as well as troubleshoot those hosts which aren’t sending but are supposed to be. There isn’t a Splunk resource in the company so I am trying my best to figure it as I go. (classic) The IT manager gave me a spreadsheet of hostnames and private IP addresses for all the devices which are forwarding logs. At first I thought I could run a search to just compare his list with logs received by hostname but I can’t figure that out. Here’s what I did instead. Over a 30-day search I run | metadata type=hosts index=* and I exported the results to a csv. I took the ‘hosts’ column (which was a combination of hostnames and IP addresses) from the export and did a diff against the IT managers list of hostnames/IP addresses and where it wasn’t found, presumed it had not sent logs during that time period. The inventory has about ~850 line items in total which are supposedly onboarded and I saw logs from about ~250. Obviously I am second guessing myself because of the delta. When I spot check some hostnames/IP addresses from the asset inventory spreadsheet from IT in Splunk, there are some that return no results, some that is just DNS or FW traffic from that server (so needs onboarding to get server logs) but others where I get results where the ‘host’ field is a cloud appliance (like Meraki) and the hostname or IP matches to other fields such as ‘dvc_up’, ‘deviceName’ or ‘dvc’ fields. This is really confusing the heck out of me and making me question if there is a better way. So, is there? How do you normally audit and verify that your logs are still being received into your Splunk instance? Thanks so much for your help and looking forward to learning!
... View more