This started out as a question, but is now just an FYI.  Similar to this post, this week I received a old vulnerability notice from Tenable about my Splunk instance.  We'd previously remediated this issue, so it was weird that it showed up again suddenly. Vulnerability details: https://packetstormsecurity.com/files/144879/Splunk-6.6.x-Local-Privilege-Escalation.html https://advisory.splunk.com/advisories/SP-CAAAP3M?301=/view/SP-CAAAP3M https://www.tenable.com/plugins/nessus/104498 The details in the articles are light, except saying to review the directions here for running Splunk as non-root: https://docs.splunk.com/Documentation/Splunk/9.1.2/Installation/RunSplunkasadifferentornon-rootuser Tenable also doesn't give details about exactly what it saw...it just says, "The current configuration of the host running Splunk was found to be vulnerable to a local privilege escalation vulnerability."   My OS is RHEL 7.x.  I'm launching Splunk using systemd with a non-root user and I have no init.d related files for Splunk.   My understanding is that launching with systemd eliminates the issue, since this way, Splunk never starts with root credentials anyway. Per Splunk's own advisory, any Splunk system is vulnerable, if: Satisfied one of the following conditions a. A Splunk init script created via $SPLUNK_HOME/bin/splunk enable boot-start –user on Splunk 6.1.x or later. b. A line with SPLUNK_OS_USER= exists in $SPLUNK_HOME/etc/splunk-launch.conf In my case, this is an old server and at one point, we did run the boot start command, which made changes to the $SPLUNK_HOME/etc/splunk-launch.conf line that sets the SPLUNK_OS_USER.  Although we had commented out the launch line, the Tenable regex is apparently broken and doesn't realize the line was disabled with a hash.  Removing the line entirely made Tenable stop reporting the vulnerability.  I assume their regex was only looking for "SPLUNK_OS_USER=<something>" so it missed the hash. Anyway, hope this helps someone.        

I need to migrate our cluster master to a new machine. It currently has these roles: Cluster Master Deployment Server Indexer License Master Search Head SHC Deployer I already migrated the License master role to the new server and it's working fine. I've been trying to follow the documentation here: https://docs.splunk.com/Documentation/Splunk/8.2.2/Indexer/Handlemanagernodefailure From what I gather, I need to copy all the files in /opt/splunk/etc/deployment-apps, /opt/splunk/etc/shcluster and /opt/splunk/etc/master-apps, plus anything that's in /opt/splunk/etc/system/local. Then add the passwords in plain text to the server.conf in the  local folder, restart Splunk on the new host and point all peers and search heads to the new master in their respective local server.conf files.  Is there anything else that needs done or would this take care of switching the cluster master entirely? And is there a specific order in which to do things?

Hello, We set HEC http input for several flows of data and related tokens, and we added ACK feature to this configuration. (following https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/AboutHECIDXAck) We work with a distributed infra, 1 Search Head, two indexers (no cluster) All was Ok with HEC but after some time we got our first error event : ERROR HttpInputDataHandler [2576842 HttpDedicatedIoThread-0] - Failed processing http input, token name=XXXX [...] reply=9, events_processed=0 INFO HttpInputDataHandler [2576844 HttpDedicatedIoThread-2] - HttpInputAckService not in healthy state. The maximum number of ACKed requests pending query has been reached. Server busy error (reply=9) leads to unavailability of HEC, but only for the token(s) where maximum number of ACKed requests pending query have been reached. Restarting the indexer is enough to get rid of the problem, but after many logs have been lost. We did some search and tried to customize some settings, but we only succeeded in delaying the 'server busy' problem (1 week to 1 month). Has anyone experienced the same problem ? How can we avoid increasing those pending query counter ? Thanks a lot for any help. etc/system/local/limits.conf [http_input] # The max number of ACK channels. max_number_of_ack_channel = 1000000 # The max number of acked requests pending query. max_number_of_acked_requests_pending_query = 10000000 # The max number of acked requests pending query per ACK channel. max_number_of_acked_requests_pending_query_per_ack_channel = 4000000 etc/system/local/server.conf [queue=parsingQueue] maxSize=10MB maxEventSize = 20MB maxIdleTime = 400 channel_cookie = AppGwAffinity (this one because we are using load balancer, so cookie is also set on LB)