How to verify splunk universal forwarder is forwarding all event log data?

[skimfl]

Good morning / afternoon, 

I am a cybersecurity professional who has been asked if there is a way to verify that splunk is capturing all the Windows Event logs. Currently the forwarder is configured to send all standard Windows log data to splunk. We utilize splunk to do domain and system cybersecurity event audits. I am confident my inputs.conf and splunk forwarders are configured properly,  but essentially trust but verify that splunk is indexing the appropriate data. 

I know splunk is in use world wide and specifically in SOCs around the world. If one were asked to verify the data is infact whole is there a way to verify and test other than manually generating events and parsing both splunk and windows event viewer periodically to verify splunk is infact receiving all data. Obviously this could be a configuration but with such a high level of concern around cybersecurity I would assume that orgs need to trust the data in splunk is accurate, but how can I verify?

Any tips?

[PickleRick · Accepted solution]

Well, short of using another tool to forward another copy of logs and compare them against the events ingested into Splunk there's no real method for 100% sure verifying that all events have been properly ingested.

But that's the common problem with all monitoring solutions. Not only the security-related ones.

What you can do to minimize probability of data ingestion problems getting unnoticed is:

- monitoring UF logs for warnings/errors

- verifying numbers of events (as someone already suggested) indexed vs. number of events on the source machine

- generating synthetic "checkpoint" events on the source and verifying if they're getting indexed properly (you could also use those checkpoint events to transmit metadata for the data volume monitoring)

It's the typical centuries-long known dillema of "who watches the watchers" or as Juvenal wrote "Quis custodiet ipsos custodes?". Every component that you have introduces some probability of possible failure and misbehaviour. You can't protect against every possible  scenario. Depending on your budget you can only mitigate some subset of them.

[skimfl]

Thank you for all of your responses. I have done what some of you have suggested. I have a Powershell script that generates a report of the number of events in a time period. That data then can be viewed in Splunk and compared. 

Thank you for all of the great advice

[chaker]

For the windows event logs, I would start with daily counts on the hosts event viewer. There are stats on the event viewer overview and summary(windows), but I would create a custom view with a custom time range that I would easily match with earliest and latest in a splunk search followed by a | stats count.

There are also .net objects and possibly powershell cmdlets that could be used as a scripted input into splunk, then you can build an alert to check that your stats count value in splunk equals the count value of your scripted input.

Something like this:

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-eventlog?view...

[gcusello]

Hi @skimfl,

you received a very curious question!

I never received it because all the customers are granted by Splunk's posizion in all indipendent observers (gartner, etc...).

Anyway, you could manually extract the daily logs from wineventlog, choosing a sample of reference servers, and check these values with splunk results.

I don't see any other answers.

I can only add that I performed some migration to Splunk from another platform and, during the parallel service, Splunk and the old platforms received the same number of logs.

Ciao.

Giuseppe