@Joetron -- From the description of the issue being seen it sounds like you might be experiencing an issue with the "Notable Event Suppression Keys" not getting updated properly. To provide some insight on what might be happening here, let me provide some information on how the "Notable Event Suppression" process works. For each "Notable Event Suppression" setup on a Search Head there is an "expire time" associated with the "Notable Event Suppression Key". When this "Expire Time" is reached the Search Head that the "Notable Event Suppression" is configured on sends a "Heartbeat" to the Search Head Captain. In this "Heartbeat" sent from the Search Head it contains the "Notable Event Suppression Key & List" which is to be distributed to the Members of the Search Head Cluster. Now the Captain does not Preemptively Push this Updated "Notable Event Suppression Key & List" it waits for the Search Head Cluster members to Check-in during their Regular "Heartbeat Check-in" to the SH Captain. So if the heartbeat_period inside of the [shclustering] has been adjusted away from the "Default" Setting of 5 seconds this can cause the following situation to arise inside of the Search Head Cluster: 1) The Notable Event Suppression Key expires 2) A New Notable Suppression Key & List are generated on the Search Head 3) The Search Head then sends a HeartBeat to the Search Head Cluster Captain and this HeartBeat contains the New Notable Event Suppression Key & List. 4) The Search Head Captain does not Preemptively send this New Notable Event Suppression Key & List to the Search Head Cluster Members 5) On the Next "Heartbeat Check-in" completed by the Search Head Cluster members the Captain then Pushes the New Notable Event Suppression Key & List. So if the heartbeat_period inside of the [shclustering] configuration Stanza inside of server.conf has been adjusted away from the Default of 5 seconds, you may be experiencing an issue where the Notable Event Suppression Key Expires just after the Search Head Cluster Members have "checked-in" to the Captain and as such the Captain waits for them to check-in again before sending the New Notable Event Suppression Key & List. Scenario: Notable Event Suppression Key expired on SH-1 at 11:02:15 am and the Search Head sends its "HeartBeat" with the Updated Notable Event Suppression Key which reaches the Captain at 11:02:17. Now SH-2 had checked in with the SH Captain at 11:02:15 and has a heartbeat_period of 30 seconds. This means that the SH-2 will not check-in to the Captain for another 27 seconds and as such this is when a Duplicate Notable Event could be generated from the Search Head Cluster Members that have not received the Updated Notable Event Suppression Key & List. You will want to double check the heartbeat_period inside of shclustering stanza inside of server.conf configuration file and if this is set higher than the Default of 5 seconds you will want to lower this setting to prevent this Scenario from arising and causing Duplicate Notable Events due to Suppression Keys not being distributed to the Search Head Cluster members. Hopefully this information helps shed some light on the "Notable Event Suppression" process. ... View more

Don't worry, there is a "work around" available to correct this known behavior. You will first need to identify how the Splunk Process is being started on your RHEL/CentOS 7 Server. The following are the steps to identify how your Splunk Service is being started (init.d or systemctl): Identify if systemctl or service is in use: service splunk status or systemctl status splunk.service Based on the Printout provided by these commands depends on what Script or Configuration file needs to be modified to resolve this type of issue. The following is a printout from a CentOS 7 Server that is using systemctl and the "OId" init.d Script: ]$ systemctl status splunk.service ● splunk.service - SYSV: Splunk indexer service Loaded: loaded (/etc/rc.d/init.d/splunk; bad; vendor preset: disabled) Active: active (exited) since Thu 2017-10-26 07:46:59 CDT; 2 weeks 6 days ago Docs: man:systemd-sysv-generator(8) As this printout is showing the use of the init.d Splunk Script the modification needed would be to: /etc/init.d/splunk and the modification would be to add the specific ulimits to the Startup Script. The following would be adding the NoFile (Number of File Descriptors) ulimit to the Splunk Startup Script. These would be placed inside of the "Starting" Function of the script: ulimit -Hn 20240 ulimit -Sn 10240 The above ulimit values would need to be added to the Start Function like so: splunk_start() { echo Starting Splunk... ulimit -Hn 20240 ulimit -Sn 10240 This will force the Linux Kernel to apply the ulimit settings for the process as its starting. Now if you are running RHEL/CentOS 7 and using a systemD unit file the modification is very similar. To identify where your systemD unit file is located for the Splunk Process you would issue the following command set: systemctl status splunk.service This will provide a printout like the one posted previously in this Post. It will provide the location of the unit file being used to load the Splunk Process. This is usually located in: /etc/systemd/system -- but could be located in a couple of different locations based on your installation and configuration of the OS. The following is an example of the unit file for Splunk on a systemD Server: ]$ systemctl status splunk.service ● splunk.service - SYSV: Splunk indexer service Loaded: loaded (/etc/systemd/system/splunk.service; bad; vendor preset: disabled) Active: active (exited) since Thu 2017-10-26 07:46:59 CDT; 2 weeks 6 days ago The following is an example of a systemD unit file prior to the ulimit modifications: [Unit] Description=Splunk indexer service Wants=network.target After=network.target Requires=thp-disable.service [Service] Type=forking Restart=on-failure User=splunk Group=splunk ExecStart=/opt/splunk/bin/splunk start ExecStop=/opt/splunk/bin/splunk stop ExecReload=/opt/splunk/bin/splunk restart StandardOutput=syslog TimeoutSec=300 PIDFile=/opt/splunk/var/run/splunk/splunkd.pid [Install] WantedBy=multi-user.target To correct the issue with ulimits not being honored for a Server that is using systemD you would want to add the following configuration options to the systemD unit file: LimitNOFILE="NUMBER" LimitNPROC="NUMBER" LimitDATA="NUMBER" LimitFSIZE="NUMBER" After making the needed modification to allow ulimits to be honored you would have a file that looks like the following: [Unit] Description=Splunk indexer service Wants=network.target After=network.target Requires=thp-disable.service [Service] Type=forking Restart=on-failure User=splunk Group=splunk ExecStart=/opt/splunk/bin/splunk start ExecStop=/opt/splunk/bin/splunk stop ExecReload=/opt/splunk/bin/splunk restart StandardOutput=syslog LimitNOFILE=65535 LimitNPROC=16384 LimitDATA=85256585 LimitFSIZE=632524157485 TimeoutSec=300 PIDFile=/opt/splunk/var/run/splunk/splunkd.pid [Install] WantedBy=multi-user.target Now of course there will be some difference between the configuration files listed in this Answers Post and the Splunk recommended ulimit sizes and I highly recommend you use the following Splunk Documentation to determine the appropriate ulimit sizes: http://docs.splunk.com/Documentation/Splunk/7.0.0/Installation/Systemrequirements#Considerations_regarding_system-wide_resource_limits_on_.2Anix_systems Be sure to verify against the version of Splunk being used as I posted the link to the "Current" Version of Splunk. With the changes mentioned in this Answers Post you will resolve any ulimit not being honored when Splunk is started during the Boot-Process. ... View more