I want to send an alert when a situation has been corrected. for example If i setup an alert for low diskspace on a host and I set this alert up to check every 15 minutes, with range back 15 minutes. I will get an alert until I correct the low disk issue on the host. but what I want is that when the problem is corrected, splunk somehow would know that it needs to send an "Alert resolved" out. The problem is I only want it to send that "Alert resolved" message out only after an alert was sent out that it was a problem. I would think that somehow a flag for an alert would need to be set when an alert is thrown so that when the condition is corrected, Splunk has a way to know that the previously it was a problem and it would send the "Alert resolved" alert Anyone know of a way to do this? Gary   ... View more

We have a requirement to send audit logs from our host servers (/var/log/audit/audit.log) to both our indexers and to a 3rd party host syslog server. I am testing with a host gary-test2.ussl.uhs with audit logs in /var/log/audit/audit.log. I have configured the universal forwarder host gary-test2.ussl.uhs to redirect all it's logs to the heavy forwarder. I would like to have the heavy forwarder send it's logs to the indexers but also a copy of all audit events to the syslog server syslogp01.ussl.uhs Here is the architecture involved with the routing. Universal Forwarder gary-test1.ussl.uhs Heavy Forwarder ussl-splkhfwt01.ussl.uhs Indexers splkidxt01.ussl.uhs splkidxt02.ussl.uhs Syslog server syslogp01.ussl.uhs (10.17.8.206) Here is how I configured the Heavy forwarder... splkhfwt01.ussl.uhs (heavy forwarder host) /opt/splunk/etc/apps/forwarder_syslog/local/props.conf [source::/var/log/audit/audit.log] TRANSFORMS-routing=troutingrsa /opt/splunk/etc/apps/forwarder_syslog/local/transforms.conf [troutingrsa] REGEX=. DEST_KEY=_SYSLOG_ROUTING FORMAT=Myroutingrsa /opt/splunk/etc/system/local/outputs.conf [tcpout] defaultGroup = default-autolb-group indexAndForward = 0 [tcpout:default-autolb-group] disabled = false server = splkidxt01.ussl.uhs:9997,ussl-splkidxt02.ussl.uhs:9997 [syslog:Myroutingrsa] server = 10.17.8.206:514 sendCookedData = false type = udp disabled = false What I am seeing is that the /var/log/audit/audit.log logs from host gary-test2.ussl.uhs are appearing in search queries on Splunk. Also those same logs are appearing on the syslog server. Here is the problems I found. Logs other than the audit.log logs from the host gary-test2.ussl.uhs are also appearing on the syslog server. I suspected that the props.conf and the transforms.conf were not doing their job so I remarked out all the settings in props.conf and transforms.conf then restarted splunk. I found that the logs continued to be sent to the syslog server which says the props.conf and transforms.conf files are having no effect. Just to be sure, in the outputs.conf, I removed the "[syslog:Myroutingrsa]" and it's settings. That then made the logs stop forwarding to the syslog server. Does anyone see what is wrong with my forwarding configuration settings? ... View more