Strategy to search entire estate for unique servers writing to audit.log

[adamnealis]

Is there a way to get the last time a host touched a file, within a certain period, e.g. earliest=-24h?

We got a request to do a "spot check" of 20 - 30 servers from a list of 720 servers, which according to someone's report run against splunk, have not written to /var/log/audit/audit.log in 24 hours.

I think it's boring to manually ssh to servers and collect ls -l ouput, and so I thought it might be nice to ask splunk which servers it has entries for in the audit.log file over the last 24h, then compare that list with the provided list in order to check how good their report is.

I have to use the GUI. First attempt (in fast mode). Tested with earliest=-1m

index=X OR index=Y earliest=-24h source=/var/log/audit/audit.log | table host | dedup host

As I watch the -24h query, I thought I'd ask if there are some more sane strategies to reduce the load.

In *nix terms I would simply connect to a server and check the ctime of the file.

The above query is just looking for all events in a 24h period, then depuping the list of servers. This seems a case of almost pure BFI. Good thing I am using index= and source=...

[tscroggins]

@adamnealis 

To see the last timestamp extracted from audit.log by host over a given time range:

| tstats max(_time) as _time where index IN (X Y) source=/var/log/audit/audit.log by host

You can add earliest and latest constraints directly to the where clause if desired.

To see that last time a host forwarded audit.log data to Splunk, summarize the _indextime field:

| tstats max(_indextime) as _time where index IN (X Y) source=/var/log/audit/audit.log by host

As with earliest and latest, you can add _index_earliest and _index_latest constraints to limit results by index time. Be mindful of the interplay between _index_earliest/_index_latest and any time value you've selected in the search UI and double check your results.