Hello deepashri: More background: Multiple forwarders only one indexer. So I would like to know how to create an alert/s that grabs the following (from ALL servers) - #1. Log failure/processing; #2. audit logs; #3. exceeds maximum audit record storage capacity (triggers at 75% capacity); #4 audit failures; #5 Admin logons and changes.
... View more
So I did a bit of research, since I could not find an out-of-the-box solution and as always there are multiple approaches:
In my opinion the prefereable way would be to use the already existing system monitoring apps for Windows (https://splunkbase.splunk.com/app/1680/) and Linux (https://splunkbase.splunk.com/app/273/). Although I'm not sure on what level they can be used directly, I think they'll give you a fair starting point. At worst you would need to write your own alerts and maybe add a software installation monitoring system (https://www.raymond.cc/blog/monitor-software-installs-remove-leftovers-install-monitor/) which log files can be indexed into Splunk.
However you could also try it from scratch. On Windows, Splunk is able to monitor the registry and you could check any changes for new software installation (https://answers.splunk.com/answers/8005/how-do-i-monitor-only-the-changes-to-windows-registry.html). For Linux there are many more solutions, since there are many more ways to install software on it. For example if you'd like to monitor installation over the APT package manager you should look into https://askubuntu.com/questions/425809/where-are-the-logs-for-apt-get and add it as an input to Splunk.
I'm sorry that I can't give you a pinpoint solution, but depending on the vast amount of variables I think that you need to invest some time adapting it to your parameters.
... View more