I'm trying to create a specific search/dashboard in Splunk Enterprise 7. We have hosts running Ubuntu 14.04 with the unattended-upgrades package installed and configured to run daily. When updates are installed that require a reboot, 2 files are created on each host: /var/run/reboot-required (with the text "*** System restart required ***") and /var/run/reboot-required.pkgs (containing the packages requesting the reboot). These files are removed upon rebooting the host.
The hosts have the universal forwarder installed and I have the 2 files mentioned above forwarded to our Splunk Enterprise server. Currently, I have this search set up as an alert:
sourcetype=reboot-required.pkgs | rex max_match=0 field=_raw "(?<Packages>[^\n]+)" | mvexpand Packages | eval _raw=Packages | stats values(Packages) as Packages dc(Packages) as "Package Count" by host
Within a given time period, I can see which hosts require a reboot and what packages are prompting the reboot.
Unfortunately, this search/dashboard does not give me the entire overview of ALL my hosts at THIS moment. For example, if I have hosts that last installed updates requiring a reboot a week ago, but have NOT since been rebooted, I would not see these hosts in the search results within the last 3 days (since there are no changes to the /var/run/reboot-required* files). However, I do expect to see no results IF the reboot-required* files do NOT exist on any host.
I want to get the latest status from the reboot-required* files on EACH host regardless of time frame. What is the best way to go about this?
Thanks in advance!
